fix(#5716): fail fast on concluded harness workflow failures by fullsend-ai-coder[bot] · Pull Request #5732 · fullsend-ai/fullsend

fullsend-ai-coder · 2026-07-29T16:03:26Z

Summary

WaitForHarnessAgent now fails fast when a recent workflow run has a terminal failure conclusion (failure, timed_out, startup_failure) instead of polling the full ~12-minute deadline. On true timeout, the error message includes diagnostic information about recent workflow runs (ID, status, conclusion, URL) for faster debugging.

Related Issue

Closes #5716

Changes

Add isTerminalFailure helper to identify failure/timed_out/startup_failure conclusions (excluding skipped/cancelled concurrency noise)
Add listRecentRunsAfter helper to fetch and filter recent workflow runs by trigger time
Add formatRunDiagnostics to build human-readable run summaries for error messages
Modify WaitForHarnessAgent to check for terminal failures on each poll iteration and include diagnostics on timeout
Add comprehensive tests covering fail-fast on each terminal conclusion, exclusion of skipped/cancelled, time filtering, and diagnostic formatting

Testing

go vet passes
All 27 package tests pass including 10 new tests
Tests cover: fail-fast on failure/timed_out/startup_failure, skipped and cancelled exclusion, runs-before-trigger-time filtering, timeout diagnostics formatting, empty diagnostics

Closes #5716

Post-script verification

Branch is not main/master (agent/5716-harness-fail-fast)
Secret scan passed (gitleaks — a486edd84e78b51ffa5fa006b624185fdc1c5965..HEAD)
PR body secret scan passed (gitleaks — no-git)
Pre-commit hooks passed (authoritative run on runner)
Tests ran inside sandbox

fullsend-ai-review · 2026-07-29T16:05:49Z

🤖 Finished Review · ✅ Success · Started 4:05 PM UTC · Completed 4:18 PM UTC
Commit: 18b23bf · View workflow run →

codecov · 2026-07-29T16:06:12Z

Codecov Report

❌ Patch coverage is 79.06977% with 9 lines in your changes missing coverage. Please review.

Files with missing lines	Patch %	Lines
...iourtest/drivers/ci/githubactions/githubactions.go	79.06%	7 Missing and 2 partials ⚠️

📢 Thoughts on this report? Let us know!

fullsend-ai-review · 2026-07-29T16:18:29Z

Review

Findings

Low

[test-gap] pkg/behaviourtest/drivers/ci/githubactions/githubactions_test.go — FakeClient.ListWorkflowRuns stores at most one WorkflowRun per workflow key (map[string]*WorkflowRun), so listHarnessRunsAfter can only ever return 0 or 1 runs in tests. The fail-fast loop iterates over all returned runs, and production ListWorkflowRuns can return many. The multi-run iteration path is untested — e.g., a scenario where the first run is skipped and the second is failure cannot be constructed with the current FakeClient.
Remediation: Consider extending FakeClient with a slice-based field (e.g., WorkflowRunsByFile map[string][]WorkflowRun) so ListWorkflowRuns can return multiple runs.
[edge-case] pkg/behaviourtest/drivers/ci/githubactions/githubactions.go:459 — listHarnessRunsAfter silently swallows ListWorkflowRuns errors and returns nil. When the API is consistently unreachable throughout the polling window, the timeout diagnostic reports "no recent workflow runs found after trigger time" — misleading when the actual problem is API unreachability. Pre-existing pattern, not a regression.

Previous run

Review

Findings

Medium

[false-positive fail-fast] pkg/behaviourtest/drivers/ci/githubactions/githubactions.go:515 — The fail-fast check in WaitForHarnessAgent iterates over all recent workflow runs from ListRecentWorkflowRuns without filtering by workflow name, agent, or any other scope. ListRecentWorkflowRuns returns runs across all workflows in the repository (it hits the /repos/{owner}/{repo}/actions/runs endpoint). If an unrelated workflow in the same repository (e.g., a CI check, a different agent's harness run) has a terminal failure conclusion, this code will abort WaitForHarnessAgent even though the harness run for the requested agent may still be in progress or may succeed. The time-window filter (after) provides some mitigation, but in repos with active CI, coincidental terminal failures within the window are plausible.
Remediation: Filter the recent runs to only those associated with the harness workflow before applying the fail-fast check. The forge.WorkflowRun struct has a Name field, and there is a ListWorkflowRuns method that filters by workflow file — either could be used to scope the check.

Low

[error-message-consistency] pkg/behaviourtest/drivers/ci/githubactions/githubactions.go:515 — The fail-fast error message uses url: %s (colon separator) while formatRunDiagnostics uses url=%s (equals separator). Minor inconsistency in metadata formatting within the same file.
Remediation: Align to url=%s to match formatRunDiagnostics.
[edge-case] pkg/behaviourtest/drivers/ci/githubactions/githubactions.go:494 — When API calls (ListRepositoryArtifacts, ListRecentWorkflowRuns) fail consistently throughout the polling window, the timeout diagnostic reports "no recent workflow runs found after trigger time" — misleading when the actual problem is API unreachability. Pre-existing pattern, not a regression introduced by this PR.

Labels: PR modifies harness agent wait logic in the behaviour test CI driver

ifireball · 2026-07-29T20:11:10Z

/fs-fix rebase the PR and fix AI review comments

fullsend-ai-coder · 2026-07-29T20:12:49Z

🤖 Finished Fix · ❌ Failure · Started 8:12 PM UTC · Completed 8:21 PM UTC
Commit: 18b23bf · View workflow run →

fullsend-ai-coder · 2026-07-29T20:21:12Z

⚠️ Post-fix script failed — Signed-off-by rejected (exit code 1)

The fix agent completed, but the post-fix script failed before finishing.

Workflow run: https://github.com/fullsend-ai/.fullsend/actions/runs/30487491366

Details:
Agent commit contains a Signed-off-by trailer. Agents must not use 'git commit -s' or append Signed-off-by trailers.
Please check the workflow logs for full details and retry with /fs-fix if appropriate.

ifireball · 2026-07-29T20:23:18Z

/fs-fix redo the last fix, make sure not to include the Signed-off-by trailer

fullsend-ai-coder · 2026-07-29T20:25:12Z

🤖 Finished Fix · ❌ Failure · Started 8:25 PM UTC · Completed 8:35 PM UTC
Commit: 18b23bf · View workflow run →

The fail-fast check in WaitForHarnessAgent previously used ListRecentWorkflowRuns, which returns runs across all workflows in the repository. This could cause false-positive fail-fast when an unrelated workflow (e.g., a CI check) had a terminal failure within the time window. Scope the check to only the harness workflow (fullsend.yaml) by using ListWorkflowRuns instead of ListRecentWorkflowRuns. Also align the fail-fast error message to use url= (equals separator) consistently with formatRunDiagnostics. Addresses review feedback on #5732

fullsend-ai-coder · 2026-07-29T20:35:15Z

⚠️ Post-fix script failed — Structured output processing failed (exit code 1)

The fix agent completed, but the post-fix script failed before finishing.

Workflow run: https://github.com/fullsend-ai/.fullsend/actions/runs/30488267454

Details:
process-fix-result.py failed with exit code 1 (bad input) for PR #5732 in fullsend-ai/fullsend
Please check the workflow logs for full details and retry with /fs-fix if appropriate.

ifireball · 2026-07-30T07:11:36Z

/fs-fix rebase and fix all remaining review issues, list what you fix or was already fixed in the output

fullsend-ai-coder · 2026-07-30T07:13:24Z

🤖 Finished Fix · ❌ Failure · Started 7:13 AM UTC · Completed 7:17 AM UTC
Commit: 49b521e · View workflow run →

fullsend-ai-coder · 2026-07-30T07:17:28Z

⚠️ Post-fix script failed — Structured output processing failed (exit code 1)

The fix agent completed, but the post-fix script failed before finishing.

Workflow run: https://github.com/fullsend-ai/.fullsend/actions/runs/30522112798

Details:
process-fix-result.py failed with exit code 1 (bad input) for PR #5732 in fullsend-ai/fullsend
Please check the workflow logs for full details and retry with /fs-fix if appropriate.

ifireball · 2026-07-30T08:14:15Z

/fs-fix rebase

fullsend-ai-coder · 2026-07-30T08:15:41Z

🤖 Finished Fix · ❌ Failure · Started 8:15 AM UTC · Completed 8:20 AM UTC
Commit: 49b521e · View workflow run →

WaitForHarnessAgent previously only checked for a success artifact, polling the full ~12-minute deadline even when the harness workflow had already concluded with a failure. This wasted CI time and produced opaque timeout messages. Add fail-fast detection: on each poll iteration, check recent workflow runs for terminal failure conclusions (failure, timed_out, startup_failure). Skipped and cancelled runs are excluded to avoid false positives from concurrency-group noise. When a terminal failure is detected, fail immediately with the run ID, conclusion, and URL. On true timeout (no artifact and no failed runs), the error now includes diagnostic information about recent workflow runs so the next CI flake is self-explanatory. Closes #5716

The fail-fast check in WaitForHarnessAgent previously used ListRecentWorkflowRuns, which returns runs across all workflows in the repository. This could cause false-positive fail-fast when an unrelated workflow (e.g., a CI check) had a terminal failure within the time window. Scope the check to only the harness workflow (fullsend.yaml) by using ListWorkflowRuns instead of ListRecentWorkflowRuns. Also align the fail-fast error message to use url= (equals separator) consistently with formatRunDiagnostics. Addresses review feedback on #5732

fullsend-ai-coder · 2026-07-30T08:20:37Z

⚠️ Post-fix script failed — Structured output processing failed (exit code 1)

The fix agent completed, but the post-fix script failed before finishing.

Workflow run: https://github.com/fullsend-ai/.fullsend/actions/runs/30525914229

Details:
process-fix-result.py failed with exit code 1 (bad input) for PR #5732 in fullsend-ai/fullsend
Please check the workflow logs for full details and retry with /fs-fix if appropriate.

ifireball · 2026-07-30T08:36:27Z

/fs-review

fullsend-ai-review · 2026-07-30T08:37:58Z

🤖 Finished Review · ✅ Success · Started 8:37 AM UTC · Completed 8:52 AM UTC
Commit: a39fca5 · View workflow run →

fullsend-ai-retro · 2026-07-30T10:39:56Z

🤖 Finished Retro · ✅ Success · Started 10:39 AM UTC · Completed 10:50 AM UTC
Commit: a39fca5 · View workflow run →

fullsend-ai-retro · 2026-07-30T10:50:04Z

Retro: PR #5732 — fail fast on concluded harness workflow failures

Overall: The code and review agents performed well on the core task. The review agent correctly identified a real medium-severity scoping bug (fail-fast not filtered to the harness workflow), and the code agent produced a solid initial implementation with comprehensive tests. However, the fix agent loop was entirely broken — all 4 fix attempts failed due to post-script infrastructure issues, not agent logic. The human ultimately had to rebase manually and re-trigger review.

Timeline

Issue test(behaviour): WaitForHarnessAgent should fail fast on failed harness runs #5716 created by @ifireball (Jul 29, 13:53 UTC)
Triage required 3–4 attempts: first two correctly identified existing draft PR test(#5707): fix missing harness agent fixtures and timeout diagnostics #5714 as addressing the issue; a sandbox podman error caused one infrastructure failure; triage succeeded after the draft PR was closed
Code agent created PR #5732 (16:03 UTC) — implementation was solid but status was reported as "Terminated" due to an OIDC token timeout when posting the completion comment (the PR was actually created successfully)
Review agent correctly flagged the fail-fast logic as not scoped to the harness workflow (medium severity) — a genuine bug that would cause false positives from unrelated workflow failures
Fix attempt 1 (20:11 UTC): Agent fixed the scoping issue but used git commit -s, violating the autonomous agent DCO exemption. Post-fix.sh guardrail correctly rejected it.
Fix attempts 2–4 (20:23, 07:11, 08:14 UTC): All failed with identical error — process-fix-result.py could not find fix-result.schema.json at relative path scripts/../schemas/ because the content-addressable resource cache does not preserve sibling directory structure. The agent succeeded in all three runs; the post-script broke.
Human intervention: @ifireball manually rebased, triggered /fs-review. Review agent approved. Human approved. Merged (10:37 UTC, Jul 30).

What went well

Review quality was strong. The review agent caught the workflow-scoping bug on first pass — a genuine medium-severity issue. On re-review after the fix, it correctly approved. The human reviewer added no additional findings.
Code agent produced good initial implementation with 10 new tests covering fail-fast, exclusion of noise conclusions, time filtering, and diagnostics formatting.
Post-fix.sh Signed-off-by guardrail worked correctly — caught the DCO violation on fix attempt 1.

What went wrong

3 fix runs wasted on the same infrastructure bug — the schema path resolution failure in process-fix-result.py blocked all progress and required human workaround.
Total time to resolution: ~21 hours — most of the delay was from fix agent failures and overnight wait.

Existing issues with new evidence

Reconcile-status step should not mark start comment as Terminated when agent already posted a success comment #4058 (reconcile-status Terminated false alarm): This retro found another instance — code run 30468064541 succeeded but was reported as "Terminated" due to OIDC token expiry during status posting.
Enable post-script-only retry when agent succeeds but post-script fails #2138 (post-script-only retry): Fix runs 2–4 are textbook examples — the agent succeeded but the post-script failed repeatedly with the same infrastructure bug; retrying only the post-script would have been sufficient.
Set FULLSEND_OUTPUT_SCHEMA to the actual schema file, not the containing directory #1516 (FULLSEND_OUTPUT_SCHEMA path): Related but distinct from the schema path bug found here — Set FULLSEND_OUTPUT_SCHEMA to the actual schema file, not the containing directory #1516 addresses the env var pointing to a directory instead of a file, while this retro's bug is about relative path resolution in process-fix-result.py under content-addressable caching.

Proposals filed

Fix relative schema path resolution in post-fix script under content-addressable cache (in fullsend-ai/agents)
Fix agent definition should explicitly prohibit git commit -s and Signed-off-by trailers (in fullsend-ai/agents)

fullsend-ai-coder Bot requested a review from a team as a code owner July 29, 2026 16:03

fullsend-ai-coder Bot added the ready-for-review Triggers review agent dispatch label Jul 29, 2026

fullsend-ai-coder Bot assigned ifireball Jul 29, 2026

This comment was marked as outdated.

Sign in to view

fullsend-ai-review Bot added requires-manual-review Review requires human judgment component/harness Agent harness, config, and skills loading labels Jul 29, 2026

fullsend-ai-coder Bot added 2 commits July 30, 2026 08:18

fullsend-ai-coder Bot force-pushed the agent/5716-harness-fail-fast branch from 49b521e to a39fca5 Compare July 30, 2026 08:20

fullsend-ai-review Bot approved these changes Jul 30, 2026

View reviewed changes

Comment thread pkg/behaviourtest/drivers/ci/githubactions/githubactions.go

fullsend-ai-review Bot added ready-for-merge All reviewers approved — ready to merge and removed requires-manual-review Review requires human judgment labels Jul 30, 2026

ifireball approved these changes Jul 30, 2026

View reviewed changes

ifireball added this pull request to the merge queue Jul 30, 2026

Merged via the queue into main with commit 99a84b9 Jul 30, 2026
23 checks passed

ifireball deleted the agent/5716-harness-fail-fast branch July 30, 2026 10:37

This was referenced Jul 30, 2026

Fix relative schema path resolution in post-fix script under content-addressable cache fullsend-ai/agents#551

Open

Fix and code agent definitions should explicitly prohibit git commit -s and Signed-off-by trailers fullsend-ai/agents#552

Open

fullsend-ai-triage Bot mentioned this pull request Aug 3, 2026

test(behaviour): fork-related harness agent timeouts across all branches #5707

Closed

fullsend-ai-retro Bot mentioned this pull request Aug 5, 2026

Restructure CONTRIBUTING.md DCO section to eliminate agent confusion about Signed-off-by fullsend-ai/agents#672

Closed

qodo-code-review Bot mentioned this pull request Aug 6, 2026

test(#5987): add behaviour coverage for code applier branch handling #5992

Merged

6 tasks

Uh oh!

Conversation

fullsend-ai-coder Bot commented Jul 29, 2026

Summary

Related Issue

Changes

Testing

Post-script verification

Uh oh!

fullsend-ai-review Bot commented Jul 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codecov Bot commented Jul 29, 2026

Codecov Report

Uh oh!

fullsend-ai-review Bot commented Jul 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Review

Findings

Low

Review

Findings

Medium

Low

Uh oh!

This comment was marked as outdated.

Uh oh!

ifireball commented Jul 29, 2026

Uh oh!

fullsend-ai-coder Bot commented Jul 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

fullsend-ai-coder Bot commented Jul 29, 2026

Uh oh!

ifireball commented Jul 29, 2026

Uh oh!

fullsend-ai-coder Bot commented Jul 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

fullsend-ai-coder Bot commented Jul 29, 2026

Uh oh!

ifireball commented Jul 30, 2026

Uh oh!

fullsend-ai-coder Bot commented Jul 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

fullsend-ai-coder Bot commented Jul 30, 2026

Uh oh!

ifireball commented Jul 30, 2026

Uh oh!

fullsend-ai-coder Bot commented Jul 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

fullsend-ai-coder Bot commented Jul 30, 2026

Uh oh!

ifireball commented Jul 30, 2026

Uh oh!

fullsend-ai-review Bot commented Jul 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

fullsend-ai-retro Bot commented Jul 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

fullsend-ai-retro Bot commented Jul 30, 2026

Retro: PR #5732 — fail fast on concluded harness workflow failures

Timeline

What went well

What went wrong

Existing issues with new evidence

Proposals filed

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

fullsend-ai-review Bot commented Jul 29, 2026 •

edited

Loading

fullsend-ai-review Bot commented Jul 29, 2026 •

edited

Loading

fullsend-ai-coder Bot commented Jul 29, 2026 •

edited

Loading

fullsend-ai-coder Bot commented Jul 29, 2026 •

edited

Loading

fullsend-ai-coder Bot commented Jul 30, 2026 •

edited

Loading

fullsend-ai-coder Bot commented Jul 30, 2026 •

edited

Loading

fullsend-ai-review Bot commented Jul 30, 2026 •

edited

Loading

fullsend-ai-retro Bot commented Jul 30, 2026 •

edited

Loading