diff --git a/docs/ADRs/0034-centralized-shim-routing-via-dispatch.md b/docs/ADRs/0034-centralized-shim-routing-via-dispatch.md index 8c5da0473b..ca048fc16b 100644 --- a/docs/ADRs/0034-centralized-shim-routing-via-dispatch.md +++ b/docs/ADRs/0034-centralized-shim-routing-via-dispatch.md @@ -146,6 +146,10 @@ The `stage` input to `dispatch.yml` becomes optional. When provided same issue/PR. In practice, only one agent should run per issue/PR at a time, and the latest event takes priority. +> **Update (2026-09):** [ADR 0098](0098-serialize-agent-runs-and-coalesce-subsequent-events.md) +> replaces automatic cancellation with serialized runs and platform-native +> pending-run coalescing. + > **Note (2026-07, [#2452](https://github.com/fullsend-ai/fullsend/issues/2452)):** Per-org > workflow-call shims now use label-aware concurrency groups with > `cancel-in-progress: false` so distinct routing labels get isolated diff --git a/docs/ADRs/0063-polling-based-work-discovery.md b/docs/ADRs/0063-polling-based-work-discovery.md index a1e115297a..58ab003434 100644 --- a/docs/ADRs/0063-polling-based-work-discovery.md +++ b/docs/ADRs/0063-polling-based-work-discovery.md @@ -351,6 +351,11 @@ still be safe to re-run (idempotent or gracefully no-op on repeat) as defense in depth — but polling does not impose a new idempotency requirement beyond what event-driven dispatch already assumes under `cancel-in-progress`. +> **Update (2026-09):** [ADR 0098](0098-serialize-agent-runs-and-coalesce-subsequent-events.md) +> replaces automatic cancellation with serialized runs and platform-native +> pending-run coalescing. Source-native locks and agent idempotency remain +> defense in depth for duplicate dispatch and side effects. + Property keys are namespaced by target repo to avoid collisions when multiple repos poll the same Jira project: @@ -480,6 +485,10 @@ required by the authorization gate. Implementations SHOULD track - **Write-then-verify races** — duplicate dispatch possible before GHA concurrency applies; mitigated by per-stage `cancel-in-progress` groups when `event_payload` projection is correct. + + > **Update (2026-09):** [ADR 0098](0098-serialize-agent-runs-and-coalesce-subsequent-events.md) + > replaces cancellation of the active run with serialized, platform-native + > pending-run coalescing for the same harness and subject. - **Work item abstraction** — harnesses and pre-scripts may need `FULLSEND_WORK_ITEM_*` plumbing for non-GitHub sources. diff --git a/docs/ADRs/0098-serialize-agent-runs-and-coalesce-subsequent-events.md b/docs/ADRs/0098-serialize-agent-runs-and-coalesce-subsequent-events.md new file mode 100644 index 0000000000..931dfd5500 --- /dev/null +++ b/docs/ADRs/0098-serialize-agent-runs-and-coalesce-subsequent-events.md @@ -0,0 +1,108 @@ +--- +title: "98. Serialize agent runs and coalesce subsequent events" +status: Accepted +relates_to: + - agent-architecture + - agent-infrastructure + - security-threat-model +topics: + - agents + - dispatch + - concurrency + - events +--- + +# 98. Serialize agent runs and coalesce subsequent events + +Date: 2026-09-02 + +## Status + +Accepted + +## Context + +Agent workflows currently use concurrency groups that cancel an in-progress run +when a later event triggers the same agent for the same issue, pull request, or +other subject. Cancellation wastes the inference and sandbox work already +performed. It also makes a burst of related events behave as competing +replacements: for example, several user comments may each cancel a run instead +of letting the agent finish and then consider the accumulated concerns. + +The dispatch architecture already gives input drivers responsibility for +producing forge-neutral `NormalizedEvent` values and gives each harness a CEL +`trigger` over those values ([ADR 0061](0061-harness-cel-dispatch.md)). The +security threat model also identifies event coalescing as a defense against +resource amplification +([security-threat-model.md](../problems/security-threat-model.md#threat-6-denial-of-service-dos--resource-exhaustion)). + +## Options + +### Continue cancelling the active run + +The newest event takes priority immediately, but completed work is discarded +and bursts can repeatedly consume tokens without producing a result. + +### Serialize every triggering event + +No event is discarded, but a burst produces redundant runs whose inputs and +effects may substantially overlap. + +### Poll for later events within `fullsend run` + +An execution loop can retrieve, authorize, and coalesce later events after each +run, with portable ordering and a deterministic follow-up limit. It requires +every input driver, including GitHub, to support polling and race-safe cursors, +and moves scheduling and repeated invocation into the execution command. + +### Preserve the active run and coalesce pending runs + +The first event starts work immediately. The execution platform retains one +pending run for the newest matching event while the agent is active, and the +next agent run reconciles all current concerns on the subject. + +## Decision + +Adopt preserve-and-coalesce scheduling for automatic agent triggers. Every event +still follows the normal input-driver normalization, authorization, harness +selection, and CEL trigger path. A matching run enters a concurrency group keyed +by harness and stable normalized-event subject. An event that fails +authorization or does not match the harness trigger creates no pending run. + +The execution platform MUST allow the active run to finish and coalesce later +matching events into one pending run representing the newest retained event. +GitHub Actions provides these semantics with a subject-scoped concurrency group, +`cancel-in-progress: false`, and its default single-pending queue +([GitHub concurrency](https://docs.github.com/en/actions/how-tos/write-workflows/choose-when-workflows-run/control-workflow-concurrency)). +All workflow layers that share responsibility for agent concurrency MUST use +compatible groups and cancellation settings. Integrations for platforms without +equivalent semantics MUST emulate them outside the agent execution process. + +Each agent run MUST reconcile the subject's current state rather than assume the +triggering event describes all outstanding work. The retained event may still +select harness overlays and provide immediate context, but `fullsend run` does +not poll for later events or invoke another run itself. A pending follow-up is a +separate platform execution and does not extend the active run's timeout window. +Explicit user or operator cancellation remains available and is outside this +policy. + +Dispatch authorization covers the event that creates a run, not every comment +or other piece of subject state the agent may read while reconciling. This +decision does not authorize agents to treat arbitrary subject content as +commands. How content provenance and actor authority constrain agent behavior is +deferred to a separate ADR; existing deterministic authorization and input +security controls remain in force. + +## Consequences + +- Agent work already in progress completes, and bursts produce at most one + follow-up run at a time, reducing token and sandbox waste. +- GitHub Actions can implement the policy without a poll driver or a new + `fullsend run` event protocol; other platforms may require extra coordination. +- Agents must inspect current subject state, while transient intermediate events + that leave no durable state may be lost. +- Trigger authorization remains deterministic, but authority over other content + discovered during reconciliation requires a future decision. +- Per-run infrastructure timeouts remain effective, but a single pending slot + does not bound consecutive runs; sustained triggering still requires rate, + cost, or loop circuit breakers. diff --git a/docs/architecture.md b/docs/architecture.md index f666a1ee80..2a1ff27633 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -272,6 +272,13 @@ The existing design principle is that [the repo is the coordinator](problems/age evaluated by `fullsend dispatch` with pluggable input/output drivers operating on a `NormalizedEvent` struct ([ADR 0061](ADRs/0061-harness-cel-dispatch.md)). +- Automatic runs are serialized per harness and event subject. Later events do + not cancel an active run. Each event still passes through authorization and + CEL routing; the execution platform coalesces matching events into one latest + pending run, and each run reconciles the subject's current state. Authority + over other comments and content discovered during reconciliation remains a + separate decision + ([ADR 0098](ADRs/0098-serialize-agent-runs-and-coalesce-subsequent-events.md)). - Per-repo **polling** complements webhook dispatch: `fullsend poll` uses poll input drivers to discover work from remote systems (Jira first), coordinates via source-native write-then-verify locks, and feeds the same dispatch pipeline @@ -303,7 +310,9 @@ The existing design principle is that [the repo is the coordinator](problems/age idempotency? (Jira polling per ADR 0063 uses entity-property locks and runner lock refresh.) - How does work assignment interact with the backlog/priority agent described in [agent-architecture.md](problems/agent-architecture.md)? -- What happens when work needs to be cancelled, retried, or reassigned? +- How should explicit cancellation, retry, and reassignment interact with the + automatic event-coalescing policy in + [ADR 0098](ADRs/0098-serialize-agent-runs-and-coalesce-subsequent-events.md)? - Does the coordinator need state (a queue, a lock, a claim system), or can it be stateless and event-driven? - When should a conversation or thread be linked to a work item (e.g. Discussion → issue) so a conversation-native agent can hand off to `/fs-code` without diff --git a/docs/problems/security-threat-model.md b/docs/problems/security-threat-model.md index cf77d4959b..99ea135def 100644 --- a/docs/problems/security-threat-model.md +++ b/docs/problems/security-threat-model.md @@ -403,7 +403,10 @@ DOS has elements that touch several existing threats: - How do we distinguish legitimate bursts of activity (e.g., a major outage generating many related bug reports) from an attack, and should rate limits be configurable per organization to account for this? - How do we handle the case where rate limiting causes legitimate high-priority issues to be delayed? - Can we implement cost estimation before committing to an agent run — predicting whether an issue will require expensive processing and routing accordingly? -- Should the event debouncing strategy from the March 31 concurrency discussion be treated as a DOS defense or purely a correctness concern? (It serves both purposes.) +- ~~Should the event debouncing strategy from the March 31 concurrency + discussion be treated as a DOS defense or purely a correctness concern?~~ It + serves both purposes; the finish-and-coalesce policy is decided in + [ADR 0098](../ADRs/0098-serialize-agent-runs-and-coalesce-subsequent-events.md). ## Cross-cutting concern: agent self-report unreliability