feat(dispatch): let a repository preserve the agent run already in flight - #7007
Open
waynesun09 wants to merge 8 commits into
Open
feat(dispatch): let a repository preserve the agent run already in flight#7007waynesun09 wants to merge 8 commits into
waynesun09 wants to merge 8 commits into
Conversation
5 tasks
Member
Author
PR Summary by QodoOptionally preserve in-flight agent stage runs
AI Description
Diagram
High-Level Assessment
Files changed (6)
|
|
🤖 Review · ❌ Terminated · Started 2:21 PM UTC · Ended 3:03 PM UTC Commit: |
Code Review by Qodo
1.
|
![qodo-code-review[bot]](https://avatars.githubusercontent.com/in/484649?s=60&v=4){kind=small}
Site previewPreview: https://9fc4712a-site.fullsend-ai.workers.dev Commit: |
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
|
🤖 Finished Review · ❌ Failure (validation failed after 2 iteration(s)) · Started 2:21 PM UTC · Completed 3:03 PM UTC Commit: Runtime: claude · Model: opus → claude-opus-4-6 · Effort: high · Cost: $6.04 |
|
🤖 Review · Commit: |
waynesun09
force-pushed
the
preserve-runs
branch
from
September 4, 2026 15:27
e56312e to
9085c4d
Compare
|
🤖 Finished Review · ✅ Success · Started 3:30 PM UTC · Completed 3:52 PM UTC Commit: Runtime: claude · Model: opus → claude-opus-4-6 · Effort: high · Cost: $5.53 |
|
Risk Assessment: elevated (3/5) DetailsPR modifies critical workflow infrastructure (reusable-dispatch.yml) and core CLI logic (run.go) with CI workflow changes and one protected path file. Elevated by active development on run.go and the behavioral nature of the concurrency toggle change. |
ReviewFindingsMedium
Low
|
![fullsend-ai-review[bot]](https://avatars.githubusercontent.com/in/3478501?s=60&v=4){kind=small}
| @@ -2787,8 +2804,49 @@ func buildRoleSlugEnvLines(h *harness.Harness) []string { | |||
| return lines | |||
| } | |||
|
|
|||
There was a problem hiding this comment.
[low] naming-convention
runFacts struct and buildRunFactsEnvLines function follow the established Go naming conventions and the build*EnvLines pattern in the file. No change needed.
| // startedAt is when the run started, in UTC. | ||
| startedAt time.Time | ||
| } | ||
|
|
There was a problem hiding this comment.
[low] api-shape
bootstrapEnv signature correctly places the required runFacts parameter before the variadic fetchEnv. No change needed.
fullsend-ai-review
Bot
added
the
requires-manual-review
Every stage job cancels the run working on a work item as soon as a second event arrives for it. The replacement pays sandbox provisioning and bootstrap before the model reads anything, then re-reads the whole item from cold, so a burst of pushes discards finished work and buys nothing: on #6513 six force-pushes produced five completed reviews of commits that were superseded within minutes. FULLSEND_PRESERVE_RUNS lets a repository choose otherwise. Unset — the default everywhere — is exactly today's behaviour. Set to "true", the run in flight finishes and the newer event waits as the single pending run, which then works from the item's current state. The agent's side of that bargain, reconciling current state rather than the state that dispatched it, is fullsend-ai/agents#1163, and is inert until the run facts in the next commit reach the sandbox. Refs #6957 Assisted-by: Claude Signed-off-by: Wayne Sun <gsun@redhat.com> The alignment test decoded cancel-in-progress as a Go bool, which an expression string cannot unmarshal into, so it moves to a yaml.Node and asserts the exact expression every stage job must carry. Signed-off-by: Wayne Sun <gsun@redhat.com>
Once a repository preserves the run in flight, that run can outlive the
state it was dispatched on, so an agent has to be able to tell what
changed underneath it before it writes its result. Export the two facts
it needs: the work item's head at run start, and the instant the run
started.
They go through bootstrapEnv rather than env.sandbox or an env/*.env
file, because .env.d files are sourced afterwards and would expand ${VAR}
host-side to an empty string, and a ${VAR} in harness env.sandbox
hard-fails ValidateRunnerEnvWith for every consumer that does not define
it. Both are reserved so a harness cannot shadow them.
An issue run exports an empty head rather than omitting the variable: the
agent side skips its re-check on an empty value, and absence and emptiness
would otherwise be indistinguishable. The consumer is
fullsend-ai/agents#1163.
Refs #6957
Assisted-by: Claude
Signed-off-by: Wayne Sun <gsun@redhat.com>
The concurrency rules describe workflow-level choices; the stage jobs in reusable-dispatch.yml now defer that one to the repository. Say so where a contributor reads the rules, and say that the expression must stay identical across stage jobs, since a mixed setting would let one role cancel while another queues on the same work item. Refs #6957 Assisted-by: Claude Signed-off-by: Wayne Sun <gsun@redhat.com>
reservedSandboxKeys stops an env.sandbox entry shadowing FULLSEND_RUN_HEAD_SHA and FULLSEND_RUN_STARTED_AT, but it says nothing about .env.d, which bootstrapEnv sources after writing them — so a harness host_files env file could overwrite either value and hand the agent a fabricated baseline. Position in the generated script is what actually protects them, which is why the ADR 0055 env.sandbox block already sits after that line for the same reason. The run facts now sit beside it. Since the ordering is behaviour rather than formatting, the line assembly moves into buildEnvScriptLines so a test can pin it; bootstrapEnv keeps the file write and upload, and host_files copying moves to uploadHostFiles. No behaviour change beyond the ordering. This does not close every route: a host_files entry whose dest is the runner's own .env replaces the file wholesale, and the same exposure applies to FULLSEND_ROLE and FULLSEND_SLUG, which are written four lines above. That gap is repo-wide and predates this change, so it is tracked separately rather than widened into this one. Assisted-by: Claude Signed-off-by: Wayne Sun <gsun@redhat.com>
ADR 0063 names per-stage cancel-in-progress as the second of two layers protecting poll dispatch from duplicate side effects. Setting FULLSEND_PRESERVE_RUNS removes that layer, leaving the poller's own lock and whatever idempotency the agent has — and the duplicate-poll case is where the trade is least favourable, because two duplicate dispatches are the same work rather than a newer state superseding an older one, so both run to completion instead of one replacing the other. Recorded where someone deciding will read it: the workflow's concurrency comment, and a cross-reference note on ADR 0063 itself, which its rules allow without rewriting an accepted decision. The default is unchanged, so that ADR's assumption still holds wherever the variable is unset. Deliberately documentation and not a code guard, and deliberately not an exemption for harness-run: all seven stage jobs stay consistent, because a job that quietly ignored the variable would be harder to reason about than the trade itself. The repository owner makes this call; our part is to make sure they make it knowing what they give up. Assisted-by: Claude Signed-off-by: Wayne Sun <gsun@redhat.com>
The concurrency test iterated six built-in stages while the workflow has seven jobs carrying the expression, so harness-run — the matrix fan-out — was unpinned and a change to it would not have been caught. It is in the map now, keyed by matrix identity rather than the event payload because a poller supplies its work item, and the failure message says that a new stage job has to be added there or its concurrency is unguarded. Converting CancelInProgress from bool to yaml.Node also lost a property nobody asked to give up: Node.Value is "true" for both a YAML boolean and the quoted string "true", and only Tag separates them, so the three literal sites would have accepted a string and the expression site would have accepted a bare boolean. Each now asserts Tag alongside Value. Both were checked against tampered fixtures rather than assumed: setting harness-run back to a literal true fails on harness-run, and quoting a reusable workflow's cancel-in-progress fails on the type. Assisted-by: Claude Signed-off-by: Wayne Sun <gsun@redhat.com>
The variable was described only in the contributor CI guide. A repository administrator deciding whether to set it reads the operations guide, which already carries the repo-variable table it belongs in. The value semantics are worth stating rather than leaving to be discovered: GitHub compares strings case-insensitively, so TRUE and True also preserve, while any other value — including 1 and yes, which someone setting a boolean flag might reasonably try — cancels. The row points at the CI guide for what preserving gives up. Assisted-by: Claude Signed-off-by: Wayne Sun <gsun@redhat.com>
runStartedAt was taken after harness resolution, base composition, token minting and env expansion — a mint call retries over the network — so activity on the work item during setup was exported to the agent as predating the run, and a preserved run could miss an update it was supposed to reconcile against. Moving the capture to the top of runAgent shrinks that window to nothing within this process, which is the whole fix available here. It is not the honest baseline and the comment says so: the run was dispatched before the process started, so the true start is the workflow run's server-side created_at, which no host clock can reach and which costs an API call to read. That is deliberately not added here, but it is recorded so the two halves do not end up disagreeing about what "run start" means — the follow-up run watcher reached the same conclusion and already uses the run record's created_at rather than its own clock. Assisted-by: Claude Signed-off-by: Wayne Sun <gsun@redhat.com>
waynesun09
force-pushed
the
preserve-runs
branch
from
September 4, 2026 19:15
9085c4d to
c6c05f8
Compare
|
🤖 Review · Started 7:17 PM UTC Commit: |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Every agent stage job cancels the run working on a work item as soon as a second event arrives for it. This lets a repository choose otherwise: the run in flight finishes, and the newer event waits as the single pending run GitHub Actions allows, which then works from the item's current state.
Default behaviour is unchanged.
FULLSEND_PRESERVE_RUNSunset, which is everywhere today, still cancels.Scope
In: the concurrency expression on the seven
reusable-dispatch.ymlstage jobs, and the two run facts an agent needs to tell what changed underneath it (FULLSEND_RUN_HEAD_SHA,FULLSEND_RUN_STARTED_AT), exported throughbootstrapEnvafter.env.dis sourced so a harness env file cannot replace them. Note they are not fully protected:reservedSandboxKeysguardsenv.sandboxonly, and an unvalidatedhost_filesdestination can still overwrite the runner's env file. That gap is pre-existing and repo-wide —FULLSEND_ROLEandFULLSEND_SLUGsit above the same line — and is tracked separately in #7010 rather than fixed here.Not in, deliberately: anything that talks to a running agent. No watcher, no receipt, no
/fs-steer, no runtime changes. When nothing steers, the run queued behind simply does the work, so there is nothing to skip and no receipt to get wrong. That half is #6959, which is stacked on this branch and is not a prerequisite for this one.The other half of the bargain is fullsend-ai/agents#1163: once the run in flight is preserved, it can outlive the state it was dispatched on, so the agent re-checks the work item once before writing its result. It skips that check when these variables are empty, so the two merge in either order.
Why
A restart is not only tokens. It pays sandbox provisioning and bootstrap before the model reads anything, then re-reads the whole item from cold. On #6513, six force-pushes produced five completed reviews of commits that were superseded within minutes, none of which led to a change.
Relation to ADR 0098 (#6909)
That ADR proposes preserve-and-coalesce as policy. It is open, and it and this change were written in parallel against the same feedback rather than against each other, so this is not offered as its implementation. If it is accepted, the natural follow-up is to drop the variable and make the behaviour unconditional.
Testing
make lintpasses (pre-commit over the full range)runfacts_test.gocovers the exports, UTC normalisation, quote escaping, the empty head on an issue run, and thatGITHUB_SHAis never used as a fallbackEvery commit builds and vets on its own.
Checklist
Refs #6957
Review round
A Codex gpt-5.6-sol pass (one reviewer, not the full squad) produced five findings, all confirmed and addressed:
.env.d; they now export below that line, beside theenv.sandboxblock that already sits there for the same reason. The general reserved-key gap is reservedSandboxKeys only guards env.sandbox, not .env.d or host_files dest #7010.cancel-in-progressduplicate-dispatch mitigation that accepted ADR 0063 relies on, leaving the Jira lock and agent idempotency. The duplicate-poll case is where preserving is least defensible, because two duplicate dispatches are the same work rather than a newer state superseding an older one. All seven jobs stay consistent; an exemption forharness-runwas considered and rejected, since a job quietly ignoring the variable is harder to reason about than the trade.FULLSEND_PRESERVE_RUNSis documented for operators in the repo-variable table, including that the comparison is case-insensitive:TRUEandTruepreserve, while1andyescancel.created_atrather than any host clock inside the process, which is noted in the code.The export ordering is now load-bearing, so
bootstrapEnv's script assembly was split into a testable function and pinned by a test, rather than being asserted only by a comment.Rebased onto current main.
functional-testswas red here for a reason outside this change: main allowlists**/claude.exeon the Vertex egress profile and this branch predated that, so OPA denied the agent binary and every triage eval case failed at zero cost. The rebase picks that fix up. Thee2ered is separate and unrelated, a pool-repo race inTestAdminInstallUninstall(422 expected head sha didn't match current head ref).