Gateway operator runs can timeout after emitting valid output

[marcusrbrown]

Problem

A web operator run can emit valid output and then end as FAILED when the Gateway hard run timeout fires before OpenCode emits session.idle.

Observed run:

• runId: 64c64589-7da9-48e1-893e-6c2585668fbd
• entityRef: marcusrbrown/systematic
• surface: web
• stream advertised contractVersion: "1.5.0"
• stream emitted normal output frames through final output seq:52
• stream then emitted terminal status:
  • phase: "FAILED"
  • status: "failed"

Gateway logs for the run show this was a timeout, not dashboard contract drift:

• run-core: stream ended due to timeout signal
• run: execution failed
• kind: "timeout"
• err: "Run timed out: event stream aborted by timeout signal"

The dashboard now accepts/rendered the 1.5.0 stream and production serves the corrected runtime, so this remaining failure belongs to Gateway/operator execution policy.

Source findings

Relevant current behavior:

• packages/gateway/src/config.ts parses GATEWAY_RUN_TIMEOUT_MS, defaulting to 600000 ms.
• packages/gateway/src/execute/run.ts computes remaining budget from run start time, creates AbortSignal.timeout(remainingBudgetMs), and passes it into runOpenCodeCore(...).
• packages/gateway/src/execute/run.ts derives approval deadlines from the same remaining budget; approvals are capped below the hard abort.
• packages/gateway/src/execute/run-core.ts treats an abort after output but before session.idle as RunCoreError('timeout').
• The SSE manager exposes terminal status and output frames; there is no separate operator-facing error-detail event today.

Infra finding:

• marcusrbrown/infra currently exposes WORKSPACE_OPENCODE_READY_TIMEOUT_MS, but does not expose or document GATEWAY_RUN_TIMEOUT_MS in the Gateway deploy flow.
• Changing the Gateway run timeout policy likely belongs in this repo first; infra can then expose the knob if needed.

Why this matters

The operator sees useful output, but the run still lands in a terminal failure state because the current hard timeout is elapsed-wall-clock based and requires session.idle before success. Longer systematic/onboarding-style runs can therefore look successful from output, then fail at the Gateway lifecycle layer.

Candidate directions

A dedicated implementation session should evaluate one of these rather than patching the dashboard:

1. Raise or make configurable the Gateway run timeout for operator/web runs.
2. Keep a hard cap, but make timeout activity-aware so active output/progress is not aborted solely for exceeding 10 minutes.
3. Add an explicit resumable/continue path before aborting long-running operator work.
4. Split human approval wait time from compute/output time so approval latency does not consume the same execution budget.
5. Improve operator-facing failure detail if timeout remains the correct outcome, without leaking sensitive run internals.

Constraints

• Do not remove the hard timeout entirely; hung streams must still release slots and fail closed.
• Preserve session.idle as the success signal unless a deliberate replacement is designed.
• Preserve approval deadline safety, including Discord interaction-token limits.
• Keep operator-visible error detail sanitized.
• If a new config knob is added, document the infra follow-up required to expose it in deployment.

Acceptance criteria

• Long-running web operator runs that continue producing output do not fail only because the existing 10-minute wall-clock budget elapsed, unless that remains an explicit product decision.
• Timeout behavior remains bounded and releases all Gateway resources.
• Timeout/failure copy makes the operator state understandable without exposing secrets or internal paths.
• Tests cover output-before-timeout, stream-ended-before-idle, approval-deadline interaction, and configured timeout behavior.

[fro-bot]

Triage: Gateway operator runs can timeout after emitting valid output

Summary

A web operator run can stream valid output through its final frame and still terminate as FAILED when the Gateway's hard wall-clock timeout fires before OpenCode emits session.idle. The reported run (runId 64c64589-…, entityRef marcusrbrown/systematic, contractVersion 1.5.0) confirms this: output advanced through seq:52, then the terminal frame reported phase: FAILED / status: failed, and Gateway logs show kind: "timeout" / Run timed out: event stream aborted by timeout signal — not contract drift. With the dashboard now accepting the 1.5.0 stream, this is squarely a Gateway/operator execution-policy problem.

Root cause (confirmed against current main)

The timeout is a single elapsed-wall-clock budget that is activity-blind: it aborts even while output is actively streaming, and success requires reaching session.idle.

Behavior	Location
GATEWAY_RUN_TIMEOUT_MS parsed, default 600000 ms (10 min), positive-int validated	packages/gateway/src/config.ts:498-505
Default wired into RunMentionDeps.runTimeoutMs	packages/gateway/src/program.ts:550
Remaining budget = runTimeoutMs − elapsedMs; AbortSignal.timeout(remainingBudgetMs) passed into runOpenCodeCore	packages/gateway/src/execute/run.ts:480-482
Approval deadline derived from the same remaining budget (max(60s, budget/2), capped at budget−30s and 13 min)	packages/gateway/src/execute/run.ts:201-205, 485
Abort after output but before session.idle → RunCoreError('timeout') (the failing path here)	packages/gateway/src/execute/run-core.ts:544-548

The key insight: at run-core.ts:365-368 the loop appends every delta to the sink, but the timeout signal at run.ts:482 is armed once at run entry and never re-armed on activity. So a run that is productively streaming past 10 minutes is aborted identically to a hung stream. The issue's "Source findings" section matches the code exactly.

Note: the issue body says marcusrbrown/infra does not expose GATEWAY_RUN_TIMEOUT_MS. That's consistent here — the knob exists in this repo's config but is not currently surfaced in the deploy flow, so today every deployment runs at the hardcoded 10-minute default.

Reproduce

No live OpenCode/workspace is reachable from CI, so I confirmed the failure mechanism by reading the execution path rather than running a live operator stream. The deterministic reproduction already exists as a behavioral invariant in the test suite — run-core.test.ts asserts that an abort fired after output (before session.idle) throws RunCoreError('timeout'), and run.test.ts asserts the visible-output timeout copy is chosen (e.g. run.test.ts:1170, :1577). To reproduce as a regression target: drive runOpenCodeCore with a sink that has appended deltas, then trip the AbortSignal before emitting session.idle → current code yields FAILED even though output was valid. That is the exact observed run.

Proposed next steps

This needs a dedicated implementation session (not a dashboard patch). Per the issue's candidate directions and constraints (keep a hard cap, preserve session.idle as the success signal, preserve approval-deadline safety incl. Discord interaction-token limits, keep operator copy sanitized), my recommended sequencing:

1. Primary — make the hard timeout activity-aware (issue direction Dependency Dashboard #2). Replace the single AbortSignal.timeout(remainingBudgetMs) with an idle-based abort: reset a "no-activity" timer whenever the stream appends output or fires a tool event (run-core.ts:365 loop is the natural hook), while retaining an absolute ceiling so hung streams still fail closed and release the slot/lock. This directly satisfies the acceptance criterion ("runs that continue producing output do not fail only because the 10-minute wall-clock elapsed") without removing the hard cap.

2. Secondary — split approval-wait from compute budget (issue direction Action Required: Fix Renovate Configuration #4). Today approval latency consumes the same budget as compute (run.ts:485). Decouple so human approval time doesn't eat execution runway, keeping the 13-minute Discord interaction-token cap intact (computeApprovalDeadlineMs, run.ts:201-205).

3. Config + infra follow-up. If the idle-timeout introduces a new knob (e.g. an idle ceiling alongside the absolute ceiling), add it in config.ts with the same _FILE/positive-int validation pattern, and file the marcusrbrown/infra follow-up to expose it in the Gateway deploy flow (currently only WORKSPACE_OPENCODE_READY_TIMEOUT_MS is surfaced).

4. Operator-facing copy (issue direction ci(renovate): use correct preset and update reusable workflow #5). Keep the sanitized timeout message, but ensure the activity-aware path only emits a timeout when genuinely idle/over-ceiling, so the operator state is understandable. No internal paths/secrets in copy (run.ts:733-746).

5. Tests. Extend run-core.test.ts / run.test.ts to cover: output-before-timeout not failing under the idle policy, stream-ended-before-idle still failing closed, absolute-ceiling abort, approval-deadline interaction, and configured-timeout behavior — matching the acceptance criteria.

I'd start with #1 since it resolves the reported failure with the smallest blast radius and is fully covered by the existing test seams.

---

Run Summary

Field	Value
Event	issues
Repository	fro-bot/agent
Run ID	28313918159
Cache	hit
Session	ses_0f310ea85ffeSGpkvdrv47CX8s