From 04938867f035d37d07dd317c1578b9f39da3a41e Mon Sep 17 00:00:00 2001 From: Barak Korren Date: Wed, 2 Sep 2026 14:48:36 +0300 Subject: [PATCH 1/2] docs(adr): serialize agent runs and coalesce subsequent events Signed-off-by: Barak Korren Assisted-by: Codex (gpt-5.6-sol) --- .codex/environments/environment.toml | 8 ++ ...4-centralized-shim-routing-via-dispatch.md | 4 + .../ADRs/0063-polling-based-work-discovery.md | 5 + ...ent-runs-and-coalesce-subsequent-events.md | 101 ++++++++++++++++++ docs/architecture.md | 10 +- docs/problems/security-threat-model.md | 5 +- 6 files changed, 131 insertions(+), 2 deletions(-) create mode 100644 .codex/environments/environment.toml create mode 100644 docs/ADRs/0098-serialize-agent-runs-and-coalesce-subsequent-events.md diff --git a/.codex/environments/environment.toml b/.codex/environments/environment.toml new file mode 100644 index 0000000000..401bd0e6c2 --- /dev/null +++ b/.codex/environments/environment.toml @@ -0,0 +1,8 @@ +# THIS IS AUTOGENERATED. DO NOT EDIT MANUALLY +version = 1 +name = "fullsend" + +[setup] +script = ''' +cd "$CODEX_WORKTREE_PATH" +mise trust -a -y''' diff --git a/docs/ADRs/0034-centralized-shim-routing-via-dispatch.md b/docs/ADRs/0034-centralized-shim-routing-via-dispatch.md index 8c5da0473b..a982b23948 100644 --- a/docs/ADRs/0034-centralized-shim-routing-via-dispatch.md +++ b/docs/ADRs/0034-centralized-shim-routing-via-dispatch.md @@ -146,6 +146,10 @@ The `stage` input to `dispatch.yml` becomes optional. When provided same issue/PR. In practice, only one agent should run per issue/PR at a time, and the latest event takes priority. +> **Update (2026-09):** [ADR 0098](0098-serialize-agent-runs-and-coalesce-subsequent-events.md) +> replaces automatic cancellation with serialized runs and bounded follow-up +> event coalescing. + > **Note (2026-07, [#2452](https://github.com/fullsend-ai/fullsend/issues/2452)):** Per-org > workflow-call shims now use label-aware concurrency groups with > `cancel-in-progress: false` so distinct routing labels get isolated diff --git a/docs/ADRs/0063-polling-based-work-discovery.md b/docs/ADRs/0063-polling-based-work-discovery.md index a1e115297a..a815b8c8b5 100644 --- a/docs/ADRs/0063-polling-based-work-discovery.md +++ b/docs/ADRs/0063-polling-based-work-discovery.md @@ -351,6 +351,11 @@ still be safe to re-run (idempotent or gracefully no-op on repeat) as defense in depth — but polling does not impose a new idempotency requirement beyond what event-driven dispatch already assumes under `cancel-in-progress`. +> **Update (2026-09):** [ADR 0098](0098-serialize-agent-runs-and-coalesce-subsequent-events.md) +> replaces automatic cancellation with serialized runs and bounded follow-up +> event coalescing. Source-native locks and agent idempotency remain defense in +> depth for duplicate dispatch and side effects. + Property keys are namespaced by target repo to avoid collisions when multiple repos poll the same Jira project: diff --git a/docs/ADRs/0098-serialize-agent-runs-and-coalesce-subsequent-events.md b/docs/ADRs/0098-serialize-agent-runs-and-coalesce-subsequent-events.md new file mode 100644 index 0000000000..bf95cb5176 --- /dev/null +++ b/docs/ADRs/0098-serialize-agent-runs-and-coalesce-subsequent-events.md @@ -0,0 +1,101 @@ +--- +title: "98. Serialize agent runs and coalesce subsequent events" +status: Accepted +relates_to: + - agent-architecture + - agent-infrastructure + - security-threat-model +topics: + - agents + - dispatch + - concurrency + - events +--- + +# 98. Serialize agent runs and coalesce subsequent events + +Date: 2026-09-02 + +## Status + +Accepted + +## Context + +Agent workflows currently use concurrency groups that cancel an in-progress run +when a later event triggers the same agent for the same issue, pull request, or +other subject. Cancellation wastes the inference and sandbox work already +performed. It also makes a burst of related events behave as competing +replacements: for example, several user comments may each cancel a run instead +of letting the agent finish and then consider the accumulated concerns. + +The dispatch architecture already gives input drivers responsibility for +producing forge-neutral `NormalizedEvent` values and gives each harness a CEL +`trigger` over those values ([ADR 0061](0061-harness-cel-dispatch.md)). The +security threat model also identifies event coalescing as a defense against +resource amplification +([security-threat-model.md](../problems/security-threat-model.md#threat-6-denial-of-service-dos--resource-exhaustion)). + +## Options + +### Continue cancelling the active run + +The newest event takes priority immediately, but completed work is discarded +and bursts can repeatedly consume tokens without producing a result. + +### Queue every triggering event + +No event is discarded, but a burst produces redundant runs whose inputs and +effects may substantially overlap. + +### Finish the active run and coalesce later events + +The first event starts work immediately; later events are considered together +after the run, trading immediate reaction to each event for bounded, useful +follow-up work. + +## Decision + +Adopt finish-and-coalesce scheduling for automatic agent triggers. For a given +harness and normalized event subject, the first matching event starts a run. +Later events do not cancel or queue another run while that run is active. +Explicit user or operator cancellation remains available and is outside this +policy. + +`fullsend dispatch` remains responsible for normalizing and authorizing the +initial event, evaluating harness triggers, and selecting runs. It MUST pass the +initial `NormalizedEvent` to `fullsend run` as a first-class input. This becomes +the canonical event input for execution; the existing legacy `event_payload` +protocol MAY remain alongside it during migration for agents that do not yet +use CEL dispatch. + +Input drivers MUST provide an operation that accepts the event that started a +run and returns later events for the same subject as `NormalizedEvent` values. +The execution loop within `fullsend run` calls this operation after the sandbox +and runtime reach a terminal state. It applies the same platform authorization +gate used for initial dispatch before evaluating the current harness's CEL +`trigger` against each returned event. If any authorized event matches, +`fullsend run` selects the newest matching event, resolves the harness and its +CEL-guarded overlays against that event, and invokes a fresh sandbox and runtime +run. Events coalesced into that follow-up do not each receive their own run. + +`fullsend run` repeats this check after each follow-up run, subject to a +platform-enforced maximum number of consecutive runs. Reaching the maximum +stops automatic continuation and produces an observable limit-exhausted result; +it does not cancel the run that reached the limit. Drivers MUST define stable +subject identity and event ordering so the check cannot move backwards or +silently cross between subjects. + +## Consequences + +- Agent work already in progress completes, and bursts produce at most one + follow-up run at a time, reducing token and sandbox waste. +- A follow-up run sees the newest triggering context and may use different + harness overlays from the preceding run. +- Input drivers must support ordered, race-safe retrieval of later events for a + subject; supporting current GitHub-event agents therefore requires GitHub + poll drivers. +- `fullsend run` must receive the initial normalized event through a canonical + input protocol, while the legacy event payload may coexist during migration. +- Follow-ups retain dispatch authorization guarantees but are delayed and + bounded, so limit exhaustion can leave work for a later trigger. diff --git a/docs/architecture.md b/docs/architecture.md index f666a1ee80..57412ea54b 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -272,6 +272,12 @@ The existing design principle is that [the repo is the coordinator](problems/age evaluated by `fullsend dispatch` with pluggable input/output drivers operating on a `NormalizedEvent` struct ([ADR 0061](ADRs/0061-harness-cel-dispatch.md)). +- Automatic runs are serialized per harness and event subject. Later events do + not cancel an active run. Dispatch passes the initial `NormalizedEvent` to + `fullsend run`; after each execution terminates, `fullsend run` uses the input + driver to retrieve and authorize newer events, re-evaluates the harness + trigger, and starts at most one bounded follow-up from the newest match + ([ADR 0098](ADRs/0098-serialize-agent-runs-and-coalesce-subsequent-events.md)). - Per-repo **polling** complements webhook dispatch: `fullsend poll` uses poll input drivers to discover work from remote systems (Jira first), coordinates via source-native write-then-verify locks, and feeds the same dispatch pipeline @@ -303,7 +309,9 @@ The existing design principle is that [the repo is the coordinator](problems/age idempotency? (Jira polling per ADR 0063 uses entity-property locks and runner lock refresh.) - How does work assignment interact with the backlog/priority agent described in [agent-architecture.md](problems/agent-architecture.md)? -- What happens when work needs to be cancelled, retried, or reassigned? +- How should explicit cancellation, retry, and reassignment interact with the + automatic event-coalescing policy in + [ADR 0098](ADRs/0098-serialize-agent-runs-and-coalesce-subsequent-events.md)? - Does the coordinator need state (a queue, a lock, a claim system), or can it be stateless and event-driven? - When should a conversation or thread be linked to a work item (e.g. Discussion → issue) so a conversation-native agent can hand off to `/fs-code` without diff --git a/docs/problems/security-threat-model.md b/docs/problems/security-threat-model.md index cf77d4959b..99ea135def 100644 --- a/docs/problems/security-threat-model.md +++ b/docs/problems/security-threat-model.md @@ -403,7 +403,10 @@ DOS has elements that touch several existing threats: - How do we distinguish legitimate bursts of activity (e.g., a major outage generating many related bug reports) from an attack, and should rate limits be configurable per organization to account for this? - How do we handle the case where rate limiting causes legitimate high-priority issues to be delayed? - Can we implement cost estimation before committing to an agent run — predicting whether an issue will require expensive processing and routing accordingly? -- Should the event debouncing strategy from the March 31 concurrency discussion be treated as a DOS defense or purely a correctness concern? (It serves both purposes.) +- ~~Should the event debouncing strategy from the March 31 concurrency + discussion be treated as a DOS defense or purely a correctness concern?~~ It + serves both purposes; the finish-and-coalesce policy is decided in + [ADR 0098](../ADRs/0098-serialize-agent-runs-and-coalesce-subsequent-events.md). ## Cross-cutting concern: agent self-report unreliability From 4fe27fc6f45e55fd258652e9c8c99690a6f09d13 Mon Sep 17 00:00:00 2001 From: Barak Korren Date: Thu, 3 Sep 2026 14:08:11 +0300 Subject: [PATCH 2/2] docs(adr): adopt platform-native event coalescing Signed-off-by: Barak Korren Assisted-by: Codex (gpt-5.6-sol) --- .codex/environments/environment.toml | 8 -- ...4-centralized-shim-routing-via-dispatch.md | 4 +- .../ADRs/0063-polling-based-work-discovery.md | 10 ++- ...ent-runs-and-coalesce-subsequent-events.md | 87 ++++++++++--------- docs/architecture.md | 9 +- 5 files changed, 61 insertions(+), 57 deletions(-) delete mode 100644 .codex/environments/environment.toml diff --git a/.codex/environments/environment.toml b/.codex/environments/environment.toml deleted file mode 100644 index 401bd0e6c2..0000000000 --- a/.codex/environments/environment.toml +++ /dev/null @@ -1,8 +0,0 @@ -# THIS IS AUTOGENERATED. DO NOT EDIT MANUALLY -version = 1 -name = "fullsend" - -[setup] -script = ''' -cd "$CODEX_WORKTREE_PATH" -mise trust -a -y''' diff --git a/docs/ADRs/0034-centralized-shim-routing-via-dispatch.md b/docs/ADRs/0034-centralized-shim-routing-via-dispatch.md index a982b23948..ca048fc16b 100644 --- a/docs/ADRs/0034-centralized-shim-routing-via-dispatch.md +++ b/docs/ADRs/0034-centralized-shim-routing-via-dispatch.md @@ -147,8 +147,8 @@ The `stage` input to `dispatch.yml` becomes optional. When provided time, and the latest event takes priority. > **Update (2026-09):** [ADR 0098](0098-serialize-agent-runs-and-coalesce-subsequent-events.md) -> replaces automatic cancellation with serialized runs and bounded follow-up -> event coalescing. +> replaces automatic cancellation with serialized runs and platform-native +> pending-run coalescing. > **Note (2026-07, [#2452](https://github.com/fullsend-ai/fullsend/issues/2452)):** Per-org > workflow-call shims now use label-aware concurrency groups with diff --git a/docs/ADRs/0063-polling-based-work-discovery.md b/docs/ADRs/0063-polling-based-work-discovery.md index a815b8c8b5..58ab003434 100644 --- a/docs/ADRs/0063-polling-based-work-discovery.md +++ b/docs/ADRs/0063-polling-based-work-discovery.md @@ -352,9 +352,9 @@ depth — but polling does not impose a new idempotency requirement beyond what event-driven dispatch already assumes under `cancel-in-progress`. > **Update (2026-09):** [ADR 0098](0098-serialize-agent-runs-and-coalesce-subsequent-events.md) -> replaces automatic cancellation with serialized runs and bounded follow-up -> event coalescing. Source-native locks and agent idempotency remain defense in -> depth for duplicate dispatch and side effects. +> replaces automatic cancellation with serialized runs and platform-native +> pending-run coalescing. Source-native locks and agent idempotency remain +> defense in depth for duplicate dispatch and side effects. Property keys are namespaced by target repo to avoid collisions when multiple repos poll the same Jira project: @@ -485,6 +485,10 @@ required by the authorization gate. Implementations SHOULD track - **Write-then-verify races** — duplicate dispatch possible before GHA concurrency applies; mitigated by per-stage `cancel-in-progress` groups when `event_payload` projection is correct. + + > **Update (2026-09):** [ADR 0098](0098-serialize-agent-runs-and-coalesce-subsequent-events.md) + > replaces cancellation of the active run with serialized, platform-native + > pending-run coalescing for the same harness and subject. - **Work item abstraction** — harnesses and pre-scripts may need `FULLSEND_WORK_ITEM_*` plumbing for non-GitHub sources. diff --git a/docs/ADRs/0098-serialize-agent-runs-and-coalesce-subsequent-events.md b/docs/ADRs/0098-serialize-agent-runs-and-coalesce-subsequent-events.md index bf95cb5176..931dfd5500 100644 --- a/docs/ADRs/0098-serialize-agent-runs-and-coalesce-subsequent-events.md +++ b/docs/ADRs/0098-serialize-agent-runs-and-coalesce-subsequent-events.md @@ -43,59 +43,66 @@ resource amplification The newest event takes priority immediately, but completed work is discarded and bursts can repeatedly consume tokens without producing a result. -### Queue every triggering event +### Serialize every triggering event No event is discarded, but a burst produces redundant runs whose inputs and effects may substantially overlap. -### Finish the active run and coalesce later events +### Poll for later events within `fullsend run` -The first event starts work immediately; later events are considered together -after the run, trading immediate reaction to each event for bounded, useful -follow-up work. +An execution loop can retrieve, authorize, and coalesce later events after each +run, with portable ordering and a deterministic follow-up limit. It requires +every input driver, including GitHub, to support polling and race-safe cursors, +and moves scheduling and repeated invocation into the execution command. + +### Preserve the active run and coalesce pending runs + +The first event starts work immediately. The execution platform retains one +pending run for the newest matching event while the agent is active, and the +next agent run reconciles all current concerns on the subject. ## Decision -Adopt finish-and-coalesce scheduling for automatic agent triggers. For a given -harness and normalized event subject, the first matching event starts a run. -Later events do not cancel or queue another run while that run is active. +Adopt preserve-and-coalesce scheduling for automatic agent triggers. Every event +still follows the normal input-driver normalization, authorization, harness +selection, and CEL trigger path. A matching run enters a concurrency group keyed +by harness and stable normalized-event subject. An event that fails +authorization or does not match the harness trigger creates no pending run. + +The execution platform MUST allow the active run to finish and coalesce later +matching events into one pending run representing the newest retained event. +GitHub Actions provides these semantics with a subject-scoped concurrency group, +`cancel-in-progress: false`, and its default single-pending queue +([GitHub concurrency](https://docs.github.com/en/actions/how-tos/write-workflows/choose-when-workflows-run/control-workflow-concurrency)). +All workflow layers that share responsibility for agent concurrency MUST use +compatible groups and cancellation settings. Integrations for platforms without +equivalent semantics MUST emulate them outside the agent execution process. + +Each agent run MUST reconcile the subject's current state rather than assume the +triggering event describes all outstanding work. The retained event may still +select harness overlays and provide immediate context, but `fullsend run` does +not poll for later events or invoke another run itself. A pending follow-up is a +separate platform execution and does not extend the active run's timeout window. Explicit user or operator cancellation remains available and is outside this policy. -`fullsend dispatch` remains responsible for normalizing and authorizing the -initial event, evaluating harness triggers, and selecting runs. It MUST pass the -initial `NormalizedEvent` to `fullsend run` as a first-class input. This becomes -the canonical event input for execution; the existing legacy `event_payload` -protocol MAY remain alongside it during migration for agents that do not yet -use CEL dispatch. - -Input drivers MUST provide an operation that accepts the event that started a -run and returns later events for the same subject as `NormalizedEvent` values. -The execution loop within `fullsend run` calls this operation after the sandbox -and runtime reach a terminal state. It applies the same platform authorization -gate used for initial dispatch before evaluating the current harness's CEL -`trigger` against each returned event. If any authorized event matches, -`fullsend run` selects the newest matching event, resolves the harness and its -CEL-guarded overlays against that event, and invokes a fresh sandbox and runtime -run. Events coalesced into that follow-up do not each receive their own run. - -`fullsend run` repeats this check after each follow-up run, subject to a -platform-enforced maximum number of consecutive runs. Reaching the maximum -stops automatic continuation and produces an observable limit-exhausted result; -it does not cancel the run that reached the limit. Drivers MUST define stable -subject identity and event ordering so the check cannot move backwards or -silently cross between subjects. +Dispatch authorization covers the event that creates a run, not every comment +or other piece of subject state the agent may read while reconciling. This +decision does not authorize agents to treat arbitrary subject content as +commands. How content provenance and actor authority constrain agent behavior is +deferred to a separate ADR; existing deterministic authorization and input +security controls remain in force. ## Consequences - Agent work already in progress completes, and bursts produce at most one follow-up run at a time, reducing token and sandbox waste. -- A follow-up run sees the newest triggering context and may use different - harness overlays from the preceding run. -- Input drivers must support ordered, race-safe retrieval of later events for a - subject; supporting current GitHub-event agents therefore requires GitHub - poll drivers. -- `fullsend run` must receive the initial normalized event through a canonical - input protocol, while the legacy event payload may coexist during migration. -- Follow-ups retain dispatch authorization guarantees but are delayed and - bounded, so limit exhaustion can leave work for a later trigger. +- GitHub Actions can implement the policy without a poll driver or a new + `fullsend run` event protocol; other platforms may require extra coordination. +- Agents must inspect current subject state, while transient intermediate events + that leave no durable state may be lost. +- Trigger authorization remains deterministic, but authority over other content + discovered during reconciliation requires a future decision. +- Per-run infrastructure timeouts remain effective, but a single pending slot + does not bound consecutive runs; sustained triggering still requires rate, + cost, or loop circuit breakers. diff --git a/docs/architecture.md b/docs/architecture.md index 57412ea54b..2a1ff27633 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -273,10 +273,11 @@ The existing design principle is that [the repo is the coordinator](problems/age operating on a `NormalizedEvent` struct ([ADR 0061](ADRs/0061-harness-cel-dispatch.md)). - Automatic runs are serialized per harness and event subject. Later events do - not cancel an active run. Dispatch passes the initial `NormalizedEvent` to - `fullsend run`; after each execution terminates, `fullsend run` uses the input - driver to retrieve and authorize newer events, re-evaluates the harness - trigger, and starts at most one bounded follow-up from the newest match + not cancel an active run. Each event still passes through authorization and + CEL routing; the execution platform coalesces matching events into one latest + pending run, and each run reconciles the subject's current state. Authority + over other comments and content discovered during reconciliation remains a + separate decision ([ADR 0098](ADRs/0098-serialize-agent-runs-and-coalesce-subsequent-events.md)). - Per-repo **polling** complements webhook dispatch: `fullsend poll` uses poll input drivers to discover work from remote systems (Jira first), coordinates