Code agent should trace production code paths before authoring integration test assertions

[fullsend-ai-retro]

What happened

PR #5407 added behaviour test scenarios for URL-sourced harness dispatch. The code agent required 11 human-guided fix iterations across 7 days because it repeatedly wrote test assertions that contradicted actual production behavior:

• Iteration 1: Assumed the deprecated per-org .fullsend repo model instead of per-repo mode, requiring architectural correction.
• Iteration 2: Wrote an allowlist-failure scenario asserting per-agent soft-skip, but production ValidateAgentEntries hard-fails the entire config — the agent assumed behavior without reading the production code.
• Iteration 5: Disagreed with the human that the hosting repo needed agents/triage.md for ADR-0045 relative resource resolution. The human traced LoadWithBase → resolveBaseResources through the call chain and proved the agent wrong.
• Iteration 6: Only after the human read pool-org dispatch logs and traced ListTriggeredHarnesses → ResolveRegisteredPath → FetchURL → isAllowedDomain was the production FetchPolicy bug (zero-value AllowedDomains) identified — the agent never independently traced this code path despite the issue referencing internal/harness/compose.go.

Each misalignment required the human to read production source code, trace the execution path, identify the discrepancy, and report the corrected understanding in a detailed /fs-fix command. The issue body provided code-path entry points (internal/harness/compose.go → FetchAgentHarness) and the code agent had full codebase access, but assertions were written based on assumed rather than verified behavior.

What could go better

When the code agent writes integration or behaviour tests for a feature, it should trace the production code paths under test before writing assertions. In this PR, had the code agent followed FetchAgentHarness → LoadWithBase → resolveBaseResources → FetchURL (for relative resources) and ListTriggeredHarnesses → ResolveRegisteredPath (for dispatch), it would have discovered in the first iteration that: (a) URL-sourced agents need relative resources on the hosting repo per ADR-0045, (b) ValidateAgentEntries hard-fails on allowlist violations, and (c) FetchPolicy wasn't threaded through dispatch enumerate — instead of discovering these in iterations 2, 5, and 6 respectively.

This pattern is distinct from verifying root cause before fixing a bug (agents#299) — the code agent wasn't fixing a bug, it was writing tests for existing functionality. The failure mode is: writing assertions about system behavior without reading the system code. For unit tests this rarely matters (assertions are local), but for integration/behaviour tests that exercise multi-component pipelines, incorrect assumptions compound across the system boundary.

Confidence: HIGH. The evidence is unambiguous across 6 iterations where the root cause was always "test assertion didn't match production behavior, and the agent hadn't read the production code path."

Proposed change

Add guidance to the code agent definition (agents/code.md) or the code-implementation skill for integration/behaviour test authoring tasks. When the issue requests tests for a feature or code path:

1. Before writing assertions, trace the production code paths referenced in the issue (or discoverable from the feature area) through the call chain. Read the actual functions, note exit conditions, error modes, and configuration requirements.
2. For behaviour/e2e tests exercising dispatch, fetch, or resolution pipelines, trace through to the security gates (SSRF policy, allowlist validation) and configuration requirements (policy defaults, interface contracts).
3. Verify that each test assertion matches the observed production behavior, not assumed behavior. When production behavior differs from what the issue description implies, note the discrepancy and write the assertion to match production (unless the issue explicitly requests behavior change).

This is a workflow directive for integration-test-authoring tasks specifically — it doesn't apply to unit tests where the system under test is local to the changed code.

Validation criteria

On the next 3 code agent runs that produce behaviour or integration test PRs for fullsend-ai/fullsend (or similar multi-component test authoring tasks), measure whether the first implementation correctly models production behavior for the tested code paths. Success: fewer than 3 human-guided fix iterations needed to correct production-behavior mismatches per PR (compared to 6 in PR #5407). The code agent's output or reasoning trace should show evidence of reading production code paths before writing assertions.

---

Generated by retro agent from fullsend-ai/fullsend#5407

[fullsend-ai-prioritize]

RICE Priority Score: 2.4

Score breakdown

Dimension	Score	Reasoning
Reach	1	This affects internal developers who use the code agent for integration/behavior test authoring across the fullsend-ai org. While not every code agent run involves integration tests, the pattern applies to any multi-component test authoring task. Scored as one strategic customer equivalent — the internal development team experiencing repeated friction.
Impact	1.5	The documented case required 11 human-guided fix iterations across 7 days on a single PR, with 6 iterations caused by production-behavior mismatches. Reducing this to fewer than 3 iterations represents a significant efficiency gain for developer workflow on integration test PRs. The pain point is well-evidenced and directly measurable.
Confidence	0.8	High confidence. The issue is exceptionally well-documented with specific iteration-by-iteration evidence from PR #5407. The failure mode is clearly identified (assertions based on assumed rather than verified behavior), the proposed change is well-scoped (add guidance to agent definition), and validation criteria are concrete and measurable.
Effort	0.5	The proposed change is adding workflow directives to agents/code.md or the code-implementation skill — essentially well-scoped instructional text. The issue already specifies the three guidance points to add. This is a documentation/instruction change, not an architectural or code change.

Formula: (1 x 1.5 x 0.8) / 0.5 = 2.4