test(#3928): add behaviour scenarios for URL-sourced harness dispatch - #5407
Conversation
|
🤖 Finished Review · ✅ Success · Started 10:16 AM UTC · Completed 10:31 AM UTC |
Site previewPreview: https://e7288326-site.fullsend-ai.workers.dev Commit: |
Codecov Report❌ Patch coverage is 📢 Thoughts on this report? Let us know! |
ReviewFindingsMedium
Low
Previous runReviewFindingsLow
Previous run (2)ReviewFindingsLow
Previous run (3)ReviewFindingsLow
Previous run (4)ReviewFindingsLow
Previous run (5)ReviewFindingsHigh
Low
Previous run (6)ReviewFindingsLow
Labels: PR adds e2e behaviour test scenarios for URL-sourced harness dispatch |
|
/fs-fix The new step functions assume the existence of the Fix the comments by the review agent as well. |
|
🤖 Finished Fix · ✅ Success · Started 11:15 PM UTC · Completed 11:28 PM UTC |
🔧 Fix agent — iteration 1 (human-triggered)Addressed all 5 actionable review findings and the human instruction. Introduced a harness-hosting repository step (CreateRepo/DeleteRepo on SCM driver) so URL dispatch tests use a separate repo instead of assuming per-org mode. Removed dead fixture, added allowlist failure scenario, fixed naming and error wrapping. Disagreed with documentation finding (out of scope). All tests pass. Fixed (5):
Disagreed (1):
Tests: passed Decision points
Updated by fullsend fix agent |
…r URL dispatch tests - Add CreateRepo/DeleteRepo to SCM driver interface for creating test repos - Introduce "Given a harness-hosting repository" step that creates a separate repo for hosting URL-sourced harness files (instead of committing to the config repo, which assumed per-org mode) - Update givenURLSourcedCustomHarness to commit harness YAML to the hosting repo and register the URL source in the enrolled test repo's config - Add cleanup for the harness-hosting repo in CleanupScenario - Add fourth scenario exercising allowlist failure path (scope-question) - Remove dead fixture file url-harness.yaml (dead-code) - Inline slices.Contains and remove misleadingly named containsPrefix helper (naming-consistency) - Wrap yaml.Marshal error for consistency (error-wrapping-consistency) Addresses review feedback on #5407
E2E tests are runningAuthorization passed for this commit. See the E2E Tests workflow for results. |
E2E tests did not runE2E tests run automatically for org/repo members and collaborators on pull requests. For other contributors, a maintainer must add the See E2E testing guide for details. |
|
/fs-review |
|
🤖 Finished Review · ✅ Success · Started 12:08 AM UTC · Completed 12:26 AM UTC |
|
/fs-fix Diagnosis (babysit)1. Behaviour tests timed out (blocking)Run: https://github.com/fullsend-ai/fullsend/actions/runs/29879270671 Progress before cancel:
Please investigate why the URL-sourced harness never dispatches / never completes after the label event (harness-hosting repo setup, allowlist wiring for the host URL, dispatch config push, artifact fetch, etc.). Do not just bump the timeout — the hang indicates the scenario is not working. 2. High review finding — allowlist scenario mismatches production
Fix by rewriting the scenario to assert actual production behavior (entire dispatch fails / no agents run), or change production to per-agent skip if that is the intended product behavior for #3928. Prefer matching production unless the issue explicitly requires soft-skip. 3. Codecov patch coverage failing (50%, need ~80%)Missing coverage concentrated in:
Add/extend unit tests (similar to existing 4. Remaining low review nits
After pushing fixes, leave a short summary of what changed. Behaviour will need |
|
🤖 Finished Fix · ✅ Success · Started 12:40 AM UTC · Completed 12:59 AM UTC |
🔧 Fix agent — iteration 2 (human-triggered)Addressed all 4 review findings. Fixed allowlist scenario to match production ValidateAgentEntries hard-fail behavior. Fixed resetScenarioWorld to clear URL harness fields preventing cross-scenario leaks. Fixed marshalling→marshaling spelling. Added 8 new unit tests improving patch coverage for url_dispatch.go, cleanup.go, and their error paths. The first-scenario timeout requires infrastructure investigation: FetchAgentHarness uses unauthenticated HTTP GET so private hosting repos will silently fail. Fixed (4):
Tests: passed Decision points
Updated by fullsend fix agent |
- Rewrite allowlist scenario to match production ValidateAgentEntries behavior: config validation hard-fails when any URL agent is outside allowed_remote_resources, preventing all agents from dispatching (not just the offending one). - Fix resetScenarioWorld to clear URLHarnessRepoOwner and URLHarnessRepoName, preventing cross-scenario state leaks. - Fix US spelling: marshalling → marshaling to match codebase convention. - Add test coverage for URL harness repo cleanup (delete, skip, error logging), agent entry update dedup, allowlist dedup, and error paths for CreateRepo and CommitFile failures. Addresses review feedback on #5407
|
/fs-fix Scope (this iteration only)Make harness-hosting repo steps use the same concurrency/lifetime model as fork steps. Do not fix the production Confirmed diagnosisBehaviour run https://github.com/fullsend-ai/fullsend/actions/runs/30303864364 job
Fork model to mirror (do the same for host)From
Required changes
Explicitly out of scope
Done when
|
|
🤖 Finished Fix · ✅ Success · Started 9:37 AM UTC · Completed 9:46 AM UTC |
Mirror the fork lifecycle model for URL-sourced harness hosting repos: - Add resolveHostRepoName to remap logical hosting repo names using the leased test repo name (e.g. "url-harness-host" + "test-repo-07" -> "test-repo-07-url-harness-host"), eliminating the shared-tip race that caused 422 "Reference cannot be updated" errors in parallel CI. - Add hosting repo deletion to CleanupScenario with the same safety guards as fork repos (skip when fields missing or name matches enrolled repo, silently ignore NotFound). - Update docs to reflect ephemeral lifecycle matching fork repos. - Add unit tests for name resolution and cleanup registration. Addresses review feedback on #5407
🔧 Fix agent — iteration 2 (human-triggered)Made harness-hosting repos ephemeral with per-lease naming to eliminate the shared-tip race causing 422 errors. Added resolveHostRepoName (mirrors resolveForkName), CleanupScenario deletion with safety guards, updated docs, and added 9 unit tests. All tests pass. Fixed (5):
Tests: passed Updated by fullsend fix agent |
|
/fs-fix Status after host-concurrency iterationRun: https://github.com/fullsend-ai/fullsend/actions/runs/30347973275
Root cause of url-ping failureTwo layers:
Required work this iterationA. Production fix (verify / complete)Ensure dispatch uses
B. Make behaviour CI actually run the fixed binary (required for green)URL scenarios must not run against stale vendored installs. Pick the smallest durable approach consistent with existing patterns:
Do not rely on “hope the leased repo needs install.” C. DocsBriefly note in behaviour-testing.md that URL-dispatch scenarios require a vendored CLI that includes FetchPolicy-aware harness dispatch (and how ensure refreshes it). Out of scope
Done when
|
|
🤖 Finished Fix · ✅ Success · Started 10:14 AM UTC · Completed 10:29 AM UTC |
Pool repos that pass post-install validation kept stale vendored binaries from prior CI runs, causing URL-dispatch scenarios to fail silently because the old binary lacked FetchPolicy-aware harness dispatch. doEnsure now always runs github setup --vendor to push the current binary, while skipping the settle wait for already-installed repos (GitHub Actions already indexed the workflow). This ensures leased pool repos always run the binary built from the current checkout. Addresses review feedback on #5407
🔧 Fix agent — iteration 3 (human-triggered)Addressed all 3 required items from the human instruction: (A) verified the FetchPolicy production fix is complete and unit-tested, (B) changed doEnsure to always re-vendor the CLI binary so pool repos run the current checkout's binary instead of stale prior-run binaries, and (C) documented the FetchPolicy and binary freshness requirements in behaviour-testing.md. The WIF 409 flake was excluded per the human's instruction. Fixed (3):
Disagreed (1):
Tests: passed Updated by fullsend fix agent |
|
/fs-fix get coverage to 80% or above |
|
🤖 Finished Fix · ✅ Success · Started 11:03 AM UTC · Completed 11:22 AM UTC |
Add tests for newly added PR functions to meet 80% patch coverage: - scm/github: GetDefaultBranch, EnsureRepoPublic (all paths including re-verify error and still-private-after-update), ParseRepo, AddIssueLabels, CloseIssue, CommitFileToBranch, CreateChangeProposal, SubmitPullRequestReview (100% coverage for all PR-added methods) - forge/fake: UpdateRepoVisibility (Repos, CreatedRepos, not-found, error injection paths) plus error injection and thread safety entries - steps/cleanup: issue close, artifact dir removal, dummy ops clear (CleanupScenario 73.1% → 96.2%) - steps/url_dispatch: relative resource not accessible after commit (givenURLSourcedCustomHarness 93.8% → 95.4%) Addresses review feedback on #5407
🔧 Fix agent — iteration 4 (human-triggered)Added 511 lines of unit tests across 4 test files to close patch coverage gaps. Key improvements: scm/github 43.5% to 100%, fake.UpdateRepoVisibility 0% to 100%, CleanupScenario 73.1% to 96.2%, givenURLSourcedCustomHarness 93.8% to 95.4%. All tests pass with -race. Fixed (4):
Tests: passed
Decision points
Updated by fullsend fix agent |
|
🤖 Finished Retro · ✅ Success · Started 11:47 AM UTC · Completed 12:05 PM UTC |
Retro: PR #5407 — behaviour tests for URL-sourced harness dispatchPR: #5407 ( Timeline
Key Findings1. Very high rework rate (11 iterations, all human-guided). Every fix iteration was driven by detailed human diagnosis with code-path traces, CI log evidence, and explicit scope boundaries. The code agent never independently diagnosed a CI failure or traced production behavior — it executed human-prescribed fixes. The root cause pattern: the agent wrote test assertions based on assumed production behavior instead of verified behavior, requiring 6 corrections before the tests matched reality. 2. Review agent quality gap. The review bot ran 7 passes and approved 5 times. Its findings were mostly low-severity (spelling, naming, comment style). It caught 1 legitimate HIGH (allowlist scenario mismatch). However, it missed all 3 HIGH findings that waynesun09 found: no-timeout HTTP client, resource leak on 3. Production bug discovered. The most impactful outcome was discovering that Evidence for Existing Issues
Autonomy AssessmentThe Proposals filed
|
Summary
Add behaviour test scenarios for URL-sourced custom agent harness dispatch, verifying that
FetchAgentHarnessURL resolution works end-to-end in theharness-dispatchpipeline.Related Issue
Fixes #3928
Changes
e2e/behaviour/features/dispatch/url-dispatch.feature): Three scenarios covering URL-sourced harness CEL trigger dispatch, mixed URL+local harness enumeration, and graceful skip on integrity hash failure.pkg/behaviourtest/steps/url_dispatch.go):givenURLSourcedCustomHarnesscommits harness YAML to the config repo, computes SHA256 integrity hash, constructsraw.githubusercontent.comURL, and registers the agent in config with the URL source andallowed_remote_resourcesupdate. SupportsbadHashandskipAllowlistoption variants for failure testing.e2e/behaviour/fixtures/dispatch/url-harness.yaml): Reference harness YAML for URL-sourced tests.pkg/behaviourtest/steps/url_dispatch_test.go): Covers input validation, URL format construction, bad hash injection, allowlist skip, andcontainsPrefixhelper.pkg/behaviourtest/steps/registry.go): Registers new URL dispatch step definitions.Testing
go test ./pkg/behaviourtest/steps/...go vetcleanChecklist
!for breaking changes)Closes #3928
Post-script verification
agent/3928-url-dispatch-behaviour-tests)d9185cb4c47c583de659ff76e15a7ba167899d3a..HEAD)