Skip to content

fix(runtime): terminate stray sandbox processes between iterations - #6753

Merged
waynesun09 merged 3 commits into
mainfrom
sandbox-clear-stray-processes
Sep 1, 2026
Merged

fix(runtime): terminate stray sandbox processes between iterations#6753
waynesun09 merged 3 commits into
mainfrom
sandbox-clear-stray-processes

Conversation

@waynesun09

@waynesun09 waynesun09 commented Aug 29, 2026

Copy link
Copy Markdown
Member

Summary

A tool command such as nohup python3 -c 'time.sleep(300)' & started by the agent's bash tool survives the agent's normal exit inside the sandbox: pi's built-in bash kills its process group only on abort/timeout, and Claude Code's Bash behaves the same. fullsend reuses the sandbox for the next validation-retry iteration, and ClearIterationArtifacts only rm -rf'd files — so strays from iteration 1 kept running into iteration 2 (holding files open, eating CPU/memory, writing into the workspace the next iteration reads). Verified empirically on 2026-08-29 while evaluating pi sub-agents: three backgrounded commands were still alive, reparented to PID 1, after the agent process exited.

Change: a runtime-neutral sweep (internal/runtime/stray_processes.go) that every ClearIterationArtifacts (Claude Code, pi, dummy) runs before the file cleanup. The POSIX-sh snippet (golden-pinned in testdata/kill_stray_processes.sh) kills every process of the sandbox user except the exec shell and its ancestors (the channel back to the runner), its own helpers, zombies, and the sandbox keep-alive main process (sleep infinity, now a shared constant sandbox.KeepAliveCommand used by createOnce); TERM first, KILL survivors after 2 s; warn-only, never fails the iteration. Uses only ps/awk/kill/sleep/id (all in the sandbox image; no pkill/pgrep). With the podman driver the OpenShell supervisor is PID 1 as root (crates/openshell-driver-podman/src/container.rs user: "0:0"), outside the sandbox user's process view, and the ancestry walk covers other drivers; the keep-alive (observed ps -o args= = sleep infinity, ppid 1) is spared by its argv (path-qualified first token tolerated) because killing it makes the sandbox terminal on OpenShell 0.0.111+ — a consequence is that an agent-started literal sleep infinity is spared too. The sweep is serialized against the credential refreshers through the runner's sandbox lock (sandboxMu, held across ClearIterationArtifacts, the OIDC token upload and the OpenAI auth seed), so a refresh can no longer be killed mid-upload; a ps failure inside the sweep exits 3 and surfaces as a warning instead of a silent "0 killed".

Test plan

  • go test ./internal/runtime/... ./internal/sandbox/... — only the two known TestDummyRuntime_* failures that occur whenever a local OpenShell gateway is running; patch coverage: script builder 100%, killStrayProcesses 91.7%, clearStrayProcesses 100%, all three ClearIterationArtifacts 100%
  • internal/runtime/kill_stray_processes_test.sh (registered in make script-test): plain stray exits 143, TERM-ignoring stray exits 137, the test shell survives, a second sweep kills 0
  • make lint clean (shellcheck on the golden)
  • Live sandbox (localhost/fullsend-sandbox, OpenShell 0.0.116, same create flags as createOnce): nohup sleep 300 &, a TERM-ignoring sleep 300 and python3 -c 'time.sleep(300)' survived the exec; the sweep reported stray processes killed: 3 in 2 s, only sleep infinity remained, the sandbox stayed Ready, a follow-up exec worked, a second sweep reported 0
  • docs/runtimes.md gets one step in the "How a run uses the runtime" diagram; docs/contributing/runtime-implementation.md states the ClearIterationArtifacts contract (sweep first, then files; failed sweep = warning) under the interface table
  • Review round (Claude + Grok): ps failure → exit 3 + warning; batched liveness probe (one ps -p per tick); keep-alive match tolerant of a path-qualified sleep; TestKeepAliveCommandMatchesSweepExclusion; shell test asserts the fake ps is first on PATH; pi fail-open test; sweep duration logged; refresher/sweep serialization with tests (TestRefreshOIDCToken_WaitsForSandboxLock, TestReseedOpenAIAuth_WaitsForSandboxLock); the changed snippet re-verified live (3 strays killed, keep-alive spared, zombie skipped, sandbox stayed Ready)

Refs #6464 (found during the pi runtime extension evaluation).

  • Review round 2 (Claude + Grok, clean of HIGH/MEDIUM after fixes): the sandbox lock now taken through a panic-safe withSandboxLock helper at all four sites; the OpenAI seed holds it only around the atomic authSeed exec; the iteration loop reports when it has waited >5 s for the lock; the hold budget (≈45 s worst case vs the 4-minute OIDC tick) is documented next to the mutex; the liveness probe's ps -p failure is now detected (KILL the full TERM'd list, then ps -p failed + exit 3) with a shell-test case; comments corrected (/bin/bash -c upload chain, full 4-minute token window, id -u); OpenCode stub annotated; Claude-runtime fail-open test; Makefile help. go test -race on the lock/refresh tests, 11/11 shell cases, make lint clean.

  • Review round 3 (9b2fccd3, rebased on main after fix(pi): activate grep/find/ls and ship rg + fd in the sandbox image #6752): reviewer asks — docs/runtimes.md paragraph replaced by a diagram step, interface table row shortened with the contract moved to a paragraph under the table, "quiesce" renamed to the sandbox lock (sandboxMu / withSandboxLock); bot lows — withSandboxLock takes a context and abandons the wait when the run is cancelled (TestWithSandboxLock_AbandonsTheWaitWhenCancelled), sandbox.KeepAliveCommand moved into the const block. go test ./internal/cli/... green, -race on the lock tests, make lint clean.

  • Review round 4 (0af942bfd, rebased on main after docs: clarify mint role vs identity so custom agents work by default #6772): bot lows — DummyPlaybackRuntime.ClearIterationArtifacts now runs the clearStrayProcesses sweep before the file cleanup, matching ClaudeRuntime/PiRuntime/DummyRuntime (new tests: TestDummyPlaybackRuntime_ClearIterationArtifacts_SweepsStraysBeforeFiles, _SweepFailureIsNotAnError); docs/guides/dev/cli-internals.md's validation-loop pseudocode now shows the sweep as a step between iterations. go build/targeted tests/make lint-all clean.

@qodo-code-review

Copy link
Copy Markdown

PR Summary by Qodo

Terminate stray sandbox processes between runtime iterations

🐞 Bug fix 🧪 Tests 📝 Documentation 🕐 40+ Minutes

Grey Divider

AI Description

• Terminate prior-iteration sandbox processes before clearing artifacts across all reusable
 runtimes.
• Preserve exec ancestry and sandbox keep-alive; escalate surviving processes from TERM to KILL.
• Treat sweep failures as warnings and verify behavior with unit and real-shell tests.
Diagram

graph TD
  A["Retry iteration"] --> B["Runtime cleanup"] --> C["Process sweep"] --> D{"Eligible process?"}
  D -->|Yes| E["TERM then KILL"] --> G["File cleanup"] --> H["Next iteration"]
  D -->|No| F["Protected processes"] --> G
Loading
High-Level Assessment

The following are alternative approaches to this PR:

1. Recreate each sandbox
  • ➕ Provides the strongest isolation boundary between iterations
  • ➕ Avoids process-enumeration races and command-line matching
  • ➖ Adds substantial retry latency and resource churn
  • ➖ Requires restoring workspace and runtime state for every iteration
2. Kill the agent process group
  • ➕ Uses native process-group semantics
  • ➕ Avoids broad user-level process enumeration
  • ➖ Detached or reparented background processes can escape the original group
  • ➖ Does not address the observed normal-exit behavior of agent Bash tools
3. Use cgroup-scoped iteration cleanup
  • ➕ Precisely contains and terminates all iteration descendants
  • ➕ Avoids matching processes by argv
  • ➖ Requires OpenShell or container-runtime lifecycle support not exposed here
  • ➖ Introduces infrastructure coupling beyond the runtime abstraction

Recommendation: Keep the proposed user-process sweep: it directly handles detached, reparented survivors while preserving sandbox reuse and remains runtime-neutral. Sandbox recreation is safer but disproportionately expensive, process-group cleanup misses the reported failure mode, and cgroup-scoped cleanup would be preferable only if OpenShell later exposes an iteration-level containment API.

Files changed (13) +655 / -5

Bug fix (5) +207 / -5
claude.goSweep Claude sandbox processes before artifact deletion +4/-0

Sweep Claude sandbox processes before artifact deletion

• Runs shared stray-process cleanup before removing Claude output and transcript artifacts.

internal/runtime/claude.go

dummy.goApply process hygiene to dummy runtime cleanup +4/-0

Apply process hygiene to dummy runtime cleanup

• Runs the shared sweep before deleting dummy runtime output, matching real agent runtime behavior.

internal/runtime/dummy.go

pi_run.goSweep pi sandbox processes before artifact deletion +4/-2

Sweep pi sandbox processes before artifact deletion

• Runs shared stray-process cleanup before removing pi outputs, sessions, and debug logs.

internal/runtime/pi_run.go

stray_processes.goAdd runtime-neutral stray-process sweeper +183/-0

Add runtime-neutral stray-process sweeper

• Introduces a POSIX shell sweep that excludes the active exec ancestry, helper processes, zombies, and canonical keep-alive command. It sends TERM before KILL, parses the killed count, bounds execution, sanitizes diagnostics, and downgrades failures to warnings.

internal/runtime/stray_processes.go

sandbox.goCentralize the sandbox keep-alive command +12/-3

Centralize the sandbox keep-alive command

• Defines the canonical keep-alive argv and uses it during sandbox creation so process cleanup exclusions cannot drift from lifecycle configuration.

internal/sandbox/sandbox.go

Tests (6) +439 / -0
claude_test.goVerify Claude cleanup ordering +18/-0

Verify Claude cleanup ordering

• Asserts that the process sweep executes before Claude's file cleanup command.

internal/runtime/claude_test.go

dummy_test.goCover dummy sweep ordering and failure tolerance +34/-0

Cover dummy sweep ordering and failure tolerance

• Verifies the sweep precedes file deletion and that sweep failures do not prevent cleanup or fail the iteration.

internal/runtime/dummy_test.go

kill_stray_processes_test.shExercise process termination under a real shell +160/-0

Exercise process termination under a real shell

• Creates normal, TERM-ignoring, and keep-alive fixtures to verify signal escalation, exclusions, shell survival, reporting, and idempotency. A filtered ps wrapper confines the test to its own process subtree.

internal/runtime/kill_stray_processes_test.sh

pi_bootstrap_test.goVerify pi cleanup ordering +6/-0

Verify pi cleanup ordering

• Extends pi runtime coverage to assert the stray-process sweep runs before artifact deletion.

internal/runtime/pi_bootstrap_test.go

stray_processes_test.goTest sweep rendering, parsing, and warning behavior +137/-0

Test sweep rendering, parsing, and warning behavior

• Pins production shell bytes to a golden file and verifies tool constraints, keep-alive protection, signal commands, count parsing, error handling, and user-facing output.

internal/runtime/stray_processes_test.go

kill_stray_processes.shPin the rendered production sweep script +84/-0

Pin the rendered production sweep script

• Provides the golden POSIX shell snippet used to ensure real-shell tests execute exactly the same command generated in production.

internal/runtime/testdata/kill_stray_processes.sh

Documentation (1) +8 / -0
runtimes.mdDocument between-iteration process cleanup +8/-0

Document between-iteration process cleanup

• Explains why reusable sandboxes need process cleanup, which processes are protected, and why sweep failures remain warning-only.

docs/runtimes.md

Other (1) +1 / -0
MakefileRun stray-process shell coverage in script tests +1/-0

Run stray-process shell coverage in script tests

• Registers the real-shell stray-process sweep test with the repository's script-test target.

Makefile

@fullsend-ai-review

fullsend-ai-review Bot commented Aug 29, 2026

Copy link
Copy Markdown

🤖 Finished Review · ✅ Success · Started 4:22 PM UTC · Completed 4:39 PM UTC

Commit: 6be9955 · View workflow run →

Runtime: claude · Model: opus → claude-opus-4-6 · Effort: high · Cost: $7.72

@github-actions

github-actions Bot commented Aug 29, 2026

Copy link
Copy Markdown

Site preview

Preview: https://810e5229-site.fullsend-ai.workers.dev

Commit: 0af942bfde1a55d7a2ec161b2e64e6a47fc875e4

@codecov

codecov Bot commented Aug 29, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 86.20690% with 8 lines in your changes missing coverage. Please review.

Files with missing lines Patch % Lines
internal/cli/run_openai.go 78.57% 3 Missing and 3 partials ⚠️
internal/runtime/stray_processes.go 91.66% 1 Missing and 1 partial ⚠️

📢 Thoughts on this report? Let us know!

@qodo-code-review

qodo-code-review Bot commented Aug 29, 2026

Copy link
Copy Markdown

Code Review by Qodo

🐞 Bugs (0) 📘 Rule violations (0) 📜 Skill insights (0)

Grey Divider


Remediation recommended

1. Runtime guide consultation undocumented ✓ Resolved 📘 Rule violation ⛨ Security
Description
The PR changes ClearIterationArtifacts behavior across runtime backends, but the PR description
does not reference docs/contributing/runtime-implementation.md and the guide does not document the
new process-sweep ordering or warning-only failure contract. This violates the required consultation
and documentation for behavioral runtime backend changes.
Code

internal/runtime/claude.go[171]

+	clearStrayProcesses(sandbox.Exec, sandboxName, os.Stderr)
Relevance

●●● Strong

Accepted runtime precedents favor documenting behavioral contracts and implementation guidance for
backend changes.

PR-#1780
PR-#2727

ⓘ Recommendations generated based on similar findings in past PRs

Evidence
Rule 2889480 requires explicit consultation of the runtime implementation guide for backend
behavioral changes and corresponding guide updates when contracts change. The cited backend and
helper code add a sweep-before-cleanup operation with warning-only failure semantics, while the
guide's runtime cleanup section does not describe that new contract and the supplied PR description
does not reference the guide.

Rule 2889480: Consult runtime implementation guide when modifying runtime.Runtime backends
internal/runtime/claude.go[167-174]
internal/runtime/pi_run.go[572-580]
internal/runtime/dummy.go[141-148]
internal/runtime/stray_processes.go[170-183]
docs/contributing/runtime-implementation.md[113-121]

Agent prompt
The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
The runtime backends now terminate stray sandbox processes before artifact deletion, with sweep failures treated as warnings, but the runtime implementation guide does not describe this behavior.

## Issue Context
PR Compliance 2889480 requires behavioral changes to `runtime.Runtime` backends to reference and, where applicable, update `docs/contributing/runtime-implementation.md`. Update the guide for the shared ordering and failure semantics, and note in the PR description that the guide was consulted.

## Fix Focus Areas
- internal/runtime/claude.go[167-174]
- internal/runtime/pi_run.go[572-580]
- internal/runtime/dummy.go[141-148]
- internal/runtime/stray_processes.go[170-183]
- docs/contributing/runtime-implementation.md[113-121]

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools


2. Termination misses new descendants ✗ Dismissed 🐞 Bug ☼ Reliability
Description
The sweep enumerates targets only once before sending TERM, so a targeted process that forks or
restarts a worker afterward creates a process absent from targets and the final KILL pass never
touches it. Such a replacement remains in the reused sandbox and defeats the cleanup this change is
intended to guarantee.
Code

internal/runtime/stray_processes.go[R130-132]

+  for p in $targets; do
+    alive "$p" && kill -s KILL "$p" 2>/dev/null
+  done
Relevance

●● Moderate

Snapshot-based cleanup has a plausible race, but historical evidence does not show this exact
process-sweep requirement being accepted.

PR-#1982
PR-#6035

ⓘ Recommendations generated based on similar findings in past PRs

Evidence
A single ps result is converted into a fixed numeric targets list, and both signal passes
iterate only that list; there is no later enumeration capable of discovering a process created after
the snapshot.

internal/runtime/stray_processes.go[64-70]
internal/runtime/stray_processes.go[99-106]
internal/runtime/stray_processes.go[116-132]

Agent prompt
The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
The fixed pre-TERM target list cannot include processes created while the sweep is running, allowing a stray's replacement or newly forked descendant to survive.

## Issue Context
Re-enumerate eligible sandbox-user processes after TERM and during/after the grace period, preserving the sweep shell, its ancestry/helpers, and only the actual keep-alive. Add a shell test where a TERM handler launches a replacement process and verify no replacement survives.

## Fix Focus Areas
- internal/runtime/stray_processes.go[64-132]
- internal/runtime/kill_stray_processes_test.sh[70-154]

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools


3. Stale PIDs risk wrong kill ✗ Dismissed 🐞 Bug ☼ Reliability
Description
The two-second grace loop checks only whether each snapshotted PID currently exists and is
non-zombie, not whether it is still the originally selected process. If a terminated target's PID is
reused during the sweep, the replacement process is treated as a survivor and receives SIGKILL.
Code

internal/runtime/stray_processes.go[R130-131]

+  for p in $targets; do
+    alive "$p" && kill -s KILL "$p" 2>/dev/null
Relevance

●● Moderate

PID reuse is a valid reliability concern, yet it is an edge-case architectural hardening request
without close precedent.

PR-#1682
PR-#1982

ⓘ Recommendations generated based on similar findings in past PRs

Evidence
Only numeric PIDs are retained from the initial snapshot. The later alive helper checks process
state for that number alone, and the KILL loop consequently cannot distinguish the original target
from a process that subsequently acquired the same PID.

internal/runtime/stray_processes.go[64-79]
internal/runtime/stray_processes.go[99-113]
internal/runtime/stray_processes.go[116-132]

Agent prompt
The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
Numeric PIDs captured once are reused for later liveness checks and SIGKILL without verifying process identity, allowing PID reuse to redirect a signal to an unrelated process.

## Issue Context
Capture a stable per-process identity such as Linux `/proc/<pid>/stat` start time and require it to match before TERM, while polling, and before KILL. Treat a missing or changed identity as the original target having exited, and add a unit/shell fixture that simulates identity change for the same PID.

## Fix Focus Areas
- internal/runtime/stray_processes.go[64-113]
- internal/runtime/stray_processes.go[116-132]
- internal/runtime/kill_stray_processes_test.sh[44-67]

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools


View medium (1)
4. Keep-alive argv spares strays ✗ Dismissed 🐞 Bug ≡ Correctness
Description
The exemption compares only the displayed argv, so every agent-started process whose command is
exactly sleep infinity is treated as the sandbox keep-alive and survives every sweep. This
deterministically leaves a valid class of previous-iteration processes running indefinitely.
Code

internal/runtime/stray_processes.go[103]

+      if (cmd[p] == keep) continue
Relevance

●● Moderate

The identity concern is technically concrete, but no close repository precedent establishes
acceptance of this keep-alive design objection.

PR-#1780
PR-#6695

ⓘ Recommendations generated based on similar findings in past PRs

Evidence
The sweep explicitly exempts every command matching the shared string, while sandbox creation passes
that same string as the keep-alive argv; no PID or other process identity participates in the
comparison.

internal/runtime/stray_processes.go[70-104]
internal/sandbox/sandbox.go[945-951]
internal/sandbox/sandbox.go[1065-1071]

Agent prompt
The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
The sweep exempts every process whose argv equals `sleep infinity`, rather than only the sandbox's actual keep-alive process.

## Issue Context
The canonical keep-alive and an agent-started `sleep infinity` have identical argv. Use a stable identity that uniquely selects the original sandbox main process, such as recording/discovering its PID or combining argv with a reliable creation-order/start-time invariant, and add a test containing a second process with the same argv.

## Fix Focus Areas
- internal/runtime/stray_processes.go[70-104]
- internal/sandbox/sandbox.go[945-951]
- internal/runtime/kill_stray_processes_test.sh[83-90]

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools


Grey Divider

Context sources
✅ Compliance rules (platform): 62 rules
Review mode: ⚖️ Balanced

Grey Divider

Tip of the day
💡 Did you know, you can describe a rule in plain language on the Rules page and Qodo drafts it for you

More tips ↗ | Customize Qodo ↗ | Qodo docs ↗

Grey Divider

Qodo Logo

Comment thread internal/runtime/claude.go
Comment thread internal/runtime/stray_processes.go
Comment thread internal/runtime/stray_processes.go Outdated
Comment thread internal/runtime/stray_processes.go Outdated
@fullsend-ai-review fullsend-ai-review Bot added the risk/elevated PR risk: elevated label Aug 29, 2026
@fullsend-ai-review

fullsend-ai-review Bot commented Aug 29, 2026

Copy link
Copy Markdown

Risk Assessment: elevated (3/5)

Details

Anchored to prior score 3/elevated — all signals essentially unchanged. Tier 1 identical: 22 files, 1299 lines, large blast radius, CI workflow change. Tier 2 confirms extreme churn in core files (run.go 85/30d with 14 authors, run_test.go 46/12, sandbox.go 26/8). Offset by decent test ratio (0.32), 4 net-new files with clean history, additive design, and non-first-time author.

Previous run

Risk Assessment: elevated (3/5)

Details

Anchored to prior score 3/elevated — all signals essentially unchanged. Tier 1 signals identical: 19 files, ~1257 lines, large blast radius, CI workflow change. Tier 2 confirms extreme churn in core files (run.go 85 commits/30d with 14 authors, sandbox.go 26/8, run_test.go 46/12). Partially offset by decent test ratio (0.32), 4 net-new files with no prior history, additive design, and non-first-time author.

Previous run (2)

Risk Assessment: elevated (3/5)

Details

Anchored to prior score 3/elevated — all signals unchanged. Large blast radius (19 files, 1231 lines) and CI workflow change drive Tier 1 upward despite no protected/security-sensitive paths. Tier 2 reveals extreme churn in core files: run.go (86 commits/30d, 14 authors, 139 fix/reverts), sandbox.go (26/9/43). Partially offset by decent test ratio (0.32), additive design, and non-first-time author.

Previous run (3)

Risk Assessment: elevated (3/5)

Details

Elevated risk driven by Tier 2 git history: runtime and sandbox packages remain under intense active development with very high churn (15 authors, 54% fix/revert rate in 90 days). The PR introduces a new subsystem (stray process termination) in this high-churn area. Large blast radius (18 files across runtime, sandbox, cli, docs, CI) and CI workflow change add Tier 1 weight. Partially offset by decent test ratio (0.33), no protected or security-sensitive paths, additive/rollback-safe design, and well-scoped implementation. Anchored to prior score 3/elevated — signals unchanged.

Previous run (4)

Risk Assessment: elevated (3/5)

Details

Elevated risk driven by Tier 2 git history: runtime and sandbox packages are under intense active development with high churn, 12 distinct authors in 90 days, and frequent fix/revert commits. The PR introduces a new subsystem (stray process termination, 183-line new file plus shell test harness) in this high-churn area. CI workflow change (Makefile) adds Tier 1 weight. Partially offset by good test ratio (0.31), no protected paths, and well-scoped change relative to the parent issue.

@fullsend-ai-review

fullsend-ai-review Bot commented Aug 29, 2026

Copy link
Copy Markdown

Review

Findings

Low

  • [consumer-completeness] internal/runtime/opencode.go:47 — OpenCodeRuntime.ClearIterationArtifacts remains a no-op and does not call clearStrayProcesses. This is intentional and documented (Run is a stub, so ClearIterationArtifacts is unreachable in practice), but when Run is implemented, the sweep must be added or the contract will be violated. The prior finding about DummyPlaybackRuntime is resolved: it now calls clearStrayProcesses.

  • [resource-leak] internal/cli/run.go:3690 — acquireSandboxLock calls time.After(sandboxLockPoll) inside a tight loop. Each call allocates a timer that is not garbage collected until it fires. With sandboxLockPoll at 100ms and worst-case hold times of ~45s, this creates up to ~450 short-lived timers. The total memory pressure is small (tens of KB) and each timer fires quickly, so this is cosmetic rather than a correctness issue.

Previous run

Review

Findings

Low

  • [consumer-completeness] internal/runtime/dummy_playback.go:291DummyPlaybackRuntime.ClearIterationArtifacts does not call clearStrayProcesses, unlike ClaudeRuntime, PiRuntime, and DummyRuntime. The per-iteration cleanup contract documented in docs/contributing/runtime-implementation.md states "Every runtime runs the shared clearStrayProcesses sweep first." OpenCodeRuntime has an explicit exemption comment explaining it is a no-op because Run is a stub, but DummyPlaybackRuntime gets no such exemption despite having an execFn and performing sandbox execs during Run.
    Remediation: Either add clearStrayProcesses(r.execFn(), sandboxName, os.Stderr) before the rm -rf in DummyPlaybackRuntime.ClearIterationArtifacts, matching the DummyRuntime pattern, or add an explicit exemption comment like the one on OpenCodeRuntime.

  • [stale-doc] docs/guides/dev/cli-internals.md:483 — The validation loop flow diagram (Phase 1 pseudocode) shows the iteration loop without a ClearIterationArtifacts step between iterations. This was a pre-existing omission, but this PR makes it more notable: ClearIterationArtifacts now sweeps stray sandbox-user processes in addition to removing files. The PR correctly updates docs/runtimes.md and docs/contributing/runtime-implementation.md, but the cli-internals flow diagram still does not mention this step.


Next steps:

  • /fs-fix — agent addresses review findings automatically
  • /fs-fix <your instruction> — agent fixes with your specific guidance
  • Push commits directly — review re-runs automatically on push
  • /fs-fix-stop — disable automatic fix runs for this PR
Previous run (2)

Review

Findings

Low

  • [edge-case] internal/cli/run.go:3649acquireSandboxQuiesce spins (TryLock + Sleep loop) with no context-cancellation path. When the iteration-clearing caller passes a non-nil notify function, the poll loop runs until the mutex is acquired — it never checks ctx.Done(). If the run’s parent context is cancelled (SIGTERM during graceful shutdown), this goroutine cannot exit until the mutex holder finishes its sandbox exec (up to ~45 s worst case, per the documented lock-hold budget).
    Remediation: Accept a context.Context in acquireSandboxQuiesce and select on ctx.Done() alongside each Sleep iteration, returning ctx.Err() when cancelled.

  • [edge-case] internal/runtime/stray_processes.go — The awk regex sub(/^[^ \t]*\//, "", line) strips only the longest non-whitespace prefix ending in /. If a future keep-alive command’s binary lives in a path that ps -o args= renders differently from basename (e.g., busybox-style multi-call binaries where the command column is busybox sleep infinity), the comparison cmd[p] == keep would not match and the sweep would kill the keep-alive, making the sandbox terminal. Safe today because the keep-alive is sleep infinity and TestKeepAliveCommandMatchesSweepExclusion pins the constant.

  • [edge-case] internal/runtime/stray_processes.go — The survivors() function passes a comma-separated PID list ($pids) to ps -p. Some minimal/busybox ps implementations accept only space-separated lists. If the sandbox image ever uses such a ps, survivors() would always fail (rc > 1), falling back to the KILL-all path on every invocation. This is graceful degradation (not a correctness bug) — the sandbox image currently ships procps-ng which supports comma-separated lists per POSIX.

  • [code-organization] internal/sandbox/sandbox.go:949KeepAliveCommand is an exported sandbox constant placed as a standalone const outside the file’s grouped const (...) block (lines 27–65) where all other exported sandbox constants live (SandboxWorkspace, SandboxClaudeConfig, etc.). Placing it near its consumer (createOnce) is defensible, but differs from the file’s established convention of grouping exported constants at the top.

  • [stale-doc] docs/guides/dev/cli-internals.md:470 — The validation loop flow diagram omits the ClearIterationArtifacts step that runs between iterations (iteration > 1). This was a pre-existing omission, but this PR makes it more significant: ClearIterationArtifacts now sweeps stray sandbox-user processes in addition to removing files. The PR correctly updates docs/runtimes.md and docs/contributing/runtime-implementation.md, but the detailed cli-internals flow diagram — the primary reference for the run loop — still does not mention this step.


Next steps:

  • /fs-fix — agent addresses review findings automatically
  • /fs-fix <your instruction> — agent fixes with your specific guidance
  • Push commits directly — review re-runs automatically on push
  • /fs-fix-stop — disable automatic fix runs for this PR
Previous run (3)

Looks good to me

Previous run (4)

Review

Findings

Low

  • [test-adequacy] internal/runtime/pi_bootstrap_test.go:385TestPiRuntimeClearIterationArtifacts uses fakeOpenshellPi which returns empty stdout for the stray-process sweep command, causing killStrayProcesses to fail the regex match and fall through to the warning path in clearStrayProcesses. The test correctly verifies ordering (sweep before file cleanup), but only exercises the error/warning branch of the sweep — the happy path is tested separately in stray_processes_test.go. Consider updating fakeOpenshellPi to output stray processes killed: 0 for the sweep command so both ordering and the happy path are exercised together.

  • [naming-consistency] internal/sandbox/sandbox.go:948KeepAliveCommand is placed as a standalone const between effectiveReadyTimeout and Create, outside the existing grouped const (...) block (lines 27–65) where other sandbox constants (SandboxWorkspace, SandboxClaudeConfig, etc.) live. Placing it near its primary consumer (createOnce) is defensible, but differs from the file's established convention of grouping all exported constants at the top.

fullsend-ai-review[bot]

This comment was marked as outdated.

@fullsend-ai-review fullsend-ai-review Bot added the ready-for-merge All reviewers approved — ready to merge label Aug 29, 2026
@waynesun09
waynesun09 force-pushed the sandbox-clear-stray-processes branch from 6be9955 to 5a3d0ca Compare August 29, 2026 16:49
@fullsend-ai-review

fullsend-ai-review Bot commented Aug 29, 2026

Copy link
Copy Markdown

🤖 Finished Review · ✅ Success · Started 4:51 PM UTC · Completed 5:08 PM UTC

Commit: 5a3d0ca · View workflow run →

Runtime: claude · Model: opus → claude-opus-4-6 · Effort: high · Cost: $9.94

fullsend-ai-review[bot]

This comment was marked as outdated.

@fullsend-ai-review

fullsend-ai-review Bot commented Aug 29, 2026

Copy link
Copy Markdown

🤖 Finished Review · ✅ Success · Started 6:22 PM UTC · Completed 7:04 PM UTC

Commit: d2aab23 · View workflow run →

Runtime: claude · Model: opus → claude-opus-4-6 · Effort: high

fullsend-ai-review[bot]

This comment was marked as outdated.

@fullsend-ai-review fullsend-ai-review Bot removed the ready-for-merge All reviewers approved — ready to merge label Aug 29, 2026
Comment thread docs/contributing/runtime-implementation.md Outdated
Comment thread docs/runtimes.md Outdated
Comment thread internal/cli/run.go Outdated
Reviewer asks (rh-hemartin) and the two remaining bot findings:

- docs/runtimes.md: the between-iterations paragraph is replaced by one
  step in the sequence diagram ("clean up stray processes"); the page is
  a runtime chooser, not a reference.
- docs/contributing/runtime-implementation.md: the interface table row
  is the suggested short form; the ClearIterationArtifacts contract
  (sweep first, then files; failed sweep = warning; held under the
  runner's sandbox lock) is a paragraph under the table instead.
- "quiesce" is gone: sandboxQuiesceMu/withSandboxQuiesce are now
  sandboxMu/withSandboxLock (+ acquireSandboxLock, sandboxLockWarnAfter,
  sandboxLockPoll and the test names).
- withSandboxLock takes a context: a waiter whose run is shutting down
  abandons the wait with ctx.Err() instead of sitting behind the
  holder's in-flight sandbox exec (up to the ~45s hold budget). Both the
  notify and the nil-notify paths poll TryLock now. New test
  TestWithSandboxLock_AbandonsTheWaitWhenCancelled; runOIDCRefresh
  already drops errors once its context is cancelled.
- sandbox.KeepAliveCommand moved into the file's const block.

Assisted-by: Claude
Signed-off-by: Wayne Sun <gsun@redhat.com>
DummyPlaybackRuntime.ClearIterationArtifacts ran the file cleanup
only, unlike ClaudeRuntime/PiRuntime/DummyRuntime — inconsistent with
the documented per-iteration cleanup contract, since dummy-playback
execs also run in the real sandbox. Add the same clearStrayProcesses
sweep, with tests mirroring DummyRuntime's.

Also add the sweep as a step in the validation-loop pseudocode in
docs/guides/dev/cli-internals.md, which still showed the loop without
it.

Assisted-by: Claude (fix), Codex (review)
Signed-off-by: Wayne Sun <gsun@redhat.com>
@fullsend-ai-review

fullsend-ai-review Bot commented Sep 1, 2026

Copy link
Copy Markdown

🤖 Finished Review · ✅ Success · Started 9:00 PM UTC · Completed 9:36 PM UTC

Commit: 0af942b · View workflow run →

Runtime: claude · Model: opus → claude-opus-4-6 · Effort: high · Cost: $5.46

@waynesun09
waynesun09 added this pull request to the merge queue Sep 1, 2026
@fullsend-ai-review
fullsend-ai-review Bot dismissed stale reviews from themself September 1, 2026 21:36

Superseded by updated review

@fullsend-ai-review fullsend-ai-review Bot added the ready-for-merge All reviewers approved — ready to merge label Sep 1, 2026
@cgwalters

Copy link
Copy Markdown
Contributor

Compare with e.g. GitHub Agentic Workflows. There's no equivalent to this in an agent definition:

validation_loop:
  script: scripts/validate-output-schema.sh
  schema: schemas/triage-result.schema.json
  max_iterations: 2

A GH-AW agent step runs an agent to completion, then reports the result. There's hence no "process leakage from prior agent steps".

Personally what I think would be much cleaner is tell the agent about the validation steps (in my projects I like Justfile, so e.g. just build etc) in the instructions - it's way more efficient for the agent to be able to see and debug the steps and process itself!

However of course yes, one wants a deterministic check - and that's what classic CI is for. GH-AW also does allow intermixing regular actions steps and agent steps which can be used for this. (Though I think in the general case, it gets a bit complicated to synchronously wait for all action results currently with GH-AW; I'm still digging into that)

Merged via the queue into main with commit a19dcf3 Sep 1, 2026
45 checks passed
@waynesun09
waynesun09 deleted the sandbox-clear-stray-processes branch September 1, 2026 21:39
@fullsend-ai-retro

fullsend-ai-retro Bot commented Sep 1, 2026

Copy link
Copy Markdown

🤖 Finished Retro · ✅ Success · Started 9:41 PM UTC · Completed 9:51 PM UTC

Commit: 0af942b · View workflow run →

Runtime: claude · Model: opus → claude-opus-4-6 · Effort: high · Cost: $2.67

Comment thread internal/runtime/testdata/kill_stray_processes.sh
@fullsend-ai-retro

Copy link
Copy Markdown

Retro: PR #6753 — Terminate stray sandbox processes between iterations

Workflow overview: Human-authored PR by waynesun09 fixing stray processes surviving between sandbox iterations (refs #6464). 3 commits, 22 files, 1299 lines. Merged after 3 days with 5 review agent runs ($33+ in review costs), 1 human reviewer approval (rh-hemartin), and 2 post-merge architectural comments from cgwalters. Agents repo: fullsend-ai/agents@main.

Timeline

  1. Aug 29 16:20 — PR opened. Review agent run 1 APPROVED ($7.72, 2 low findings: test-adequacy gap, const placement).
  2. Aug 29 16:23 — qodo-code-review posted 4 medium findings (keep-alive argv bypass, termination race, stale PID risk, undocumented guide). Author explained all were by-design trade-offs.
  3. Aug 29 16:49 — Review agent run 2 APPROVED ($9.94, zero findings — "looks good to me").
  4. Aug 29 18:20 — Review agent run 3 posted 5 findings including a genuine context-cancellation edge-case in acquireSandboxQuiesce (no ctx.Done() path). This bug existed since the first commit but was missed by runs 1 and 2. DISMISSED when new commit pushed.
  5. Aug 31 06:54 — Human reviewer rh-hemartin left 3 comments: (a) simplify table row in runtime-implementation.md, (b) runtimes.md has too much implementation detail for a user-facing selection guide — suggest diagram step instead, (c) replace jargon "quiesce" with simpler naming. APPROVED.
  6. Sep 1 20:31 — Author pushed commit 9b2fccd addressing all review feedback with 9 detailed reply comments.
  7. Sep 1 20:32 — Review agent run on 9b2fccd terminated (likely concurrency group conflict from rapid reply comments). A second run on the same commit completed at 20:52 ($9.60), finding DummyPlaybackRuntime missing sweep call.
  8. Sep 1 20:58 — Review agent run 5 APPROVED ($5.46, 2 low findings). Final commit 0af942b.
  9. Sep 1 21:39 — Merged.

What went well

  • Complementary review coverage: The review agent found a genuine context-cancellation bug and a consumer-completeness gap (DummyPlaybackRuntime). The human reviewer caught documentation-audience mismatch and naming jargon. Together they produced better coverage than either alone.
  • Risk assessment was accurate and consistent across all 5 runs: elevated 3/5.
  • Author was thorough and responsive, addressing all feedback with clear commit-by-commit replies referencing specific changes.
  • Finding evolution worked: The DummyPlaybackRuntime gap found in run 4 was confirmed resolved in run 5.

Evidence for existing issues (no new proposals needed)

waynesun09 added a commit that referenced this pull request Sep 3, 2026
The stray-process sweep waited a fixed 2s between TERM and KILL. A
maintainer objected on #6753 that this leaves an agent no room to flush
state on SIGTERM, and that objection lands hardest on the codex steer path:
ClearIterationArtifacts sweeps leftovers from a run that is already over,
but codexSteerQueue.interrupt stops a process the runner intends to
CONTINUE, one turn of a thread it is about to resume.

The grace is now a parameter of the snippet (__GRACE_TICKS__ for the poll
loop, __GRACE_LABEL__ for its own comment). ClearIterationArtifacts keeps
2s and renders byte-for-byte what it rendered before —
testdata/kill_stray_processes.sh is unchanged, and its golden test still
passes untouched. The codex interrupt gets 10s through interruptSweep,
which is the default of the existing injectable `sweep:` field, so tests
can still replace the whole sweep.

The exec timeout scales with the grace rather than staying at a flat 15s.
That bound exists to catch a hung gateway; left fixed, raising the grace
would have meant the timeout fired during the TERM wait and the KILL pass
never ran — the sweep would have started leaking exactly the processes it
exists to remove. Default stays 15s (2s + 13s headroom), the interrupt gets
23s, and a test asserts the timeout always outlasts the grace with room to
spare.

The interrupt rendering is pinned in
testdata/kill_stray_processes_interrupt.sh the same way the default one is,
and kill_stray_processes_test.sh now takes an optional snippet path so
either can be executed under a real shell. Both were run: the interrupt
golden passes, taking ~26s against the default's ~16s because the
TERM-ignoring fixture now takes its full 10s before the KILL lands — which
is the evidence that the longer wait actually elapses and the KILL pass
still works, rather than just that the string renders. A test also
normalises the two renderings and asserts they differ ONLY in the grace, so
the process-selection logic the sandbox depends on cannot fork.

codex_steer.go now records why the grace differs there, alongside the note
that every interrupt leaves a dangling tool call in the rollout (codex logs
"Custom tool call output is missing" on the resume and tolerates it).

Assisted-by: Claude
Signed-off-by: Wayne Sun <gsun@redhat.com>
waynesun09 added a commit that referenced this pull request Sep 4, 2026
The stray-process sweep waited a fixed 2s between TERM and KILL. A
maintainer objected on #6753 that this leaves an agent no room to flush
state on SIGTERM, and that objection lands hardest on the codex steer path:
ClearIterationArtifacts sweeps leftovers from a run that is already over,
but codexSteerQueue.interrupt stops a process the runner intends to
CONTINUE, one turn of a thread it is about to resume.

The grace is now a parameter of the snippet (__GRACE_TICKS__ for the poll
loop, __GRACE_LABEL__ for its own comment). ClearIterationArtifacts keeps
2s and renders byte-for-byte what it rendered before —
testdata/kill_stray_processes.sh is unchanged, and its golden test still
passes untouched. The codex interrupt gets 10s through interruptSweep,
which is the default of the existing injectable `sweep:` field, so tests
can still replace the whole sweep.

The exec timeout scales with the grace rather than staying at a flat 15s.
That bound exists to catch a hung gateway; left fixed, raising the grace
would have meant the timeout fired during the TERM wait and the KILL pass
never ran — the sweep would have started leaking exactly the processes it
exists to remove. Default stays 15s (2s + 13s headroom), the interrupt gets
23s, and a test asserts the timeout always outlasts the grace with room to
spare.

The interrupt rendering is pinned in
testdata/kill_stray_processes_interrupt.sh the same way the default one is,
and kill_stray_processes_test.sh now takes an optional snippet path so
either can be executed under a real shell. Both were run: the interrupt
golden passes, taking ~26s against the default's ~16s because the
TERM-ignoring fixture now takes its full 10s before the KILL lands — which
is the evidence that the longer wait actually elapses and the KILL pass
still works, rather than just that the string renders. A test also
normalises the two renderings and asserts they differ ONLY in the grace, so
the process-selection logic the sandbox depends on cannot fork.

codex_steer.go now records why the grace differs there, alongside the note
that every interrupt leaves a dangling tool call in the rollout (codex logs
"Custom tool call output is missing" on the resume and tolerates it).

Assisted-by: Claude
Signed-off-by: Wayne Sun <gsun@redhat.com>
waynesun09 added a commit that referenced this pull request Sep 4, 2026
The stray-process sweep waited a fixed 2s between TERM and KILL. A
maintainer objected on #6753 that this leaves an agent no room to flush
state on SIGTERM, and that objection lands hardest on the codex steer path:
ClearIterationArtifacts sweeps leftovers from a run that is already over,
but codexSteerQueue.interrupt stops a process the runner intends to
CONTINUE, one turn of a thread it is about to resume.

The grace is now a parameter of the snippet (__GRACE_TICKS__ for the poll
loop, __GRACE_LABEL__ for its own comment). ClearIterationArtifacts keeps
2s and renders byte-for-byte what it rendered before —
testdata/kill_stray_processes.sh is unchanged, and its golden test still
passes untouched. The codex interrupt gets 10s through interruptSweep,
which is the default of the existing injectable `sweep:` field, so tests
can still replace the whole sweep.

The exec timeout scales with the grace rather than staying at a flat 15s.
That bound exists to catch a hung gateway; left fixed, raising the grace
would have meant the timeout fired during the TERM wait and the KILL pass
never ran — the sweep would have started leaking exactly the processes it
exists to remove. Default stays 15s (2s + 13s headroom), the interrupt gets
23s, and a test asserts the timeout always outlasts the grace with room to
spare.

The interrupt rendering is pinned in
testdata/kill_stray_processes_interrupt.sh the same way the default one is,
and kill_stray_processes_test.sh now takes an optional snippet path so
either can be executed under a real shell. Both were run: the interrupt
golden passes, taking ~26s against the default's ~16s because the
TERM-ignoring fixture now takes its full 10s before the KILL lands — which
is the evidence that the longer wait actually elapses and the KILL pass
still works, rather than just that the string renders. A test also
normalises the two renderings and asserts they differ ONLY in the grace, so
the process-selection logic the sandbox depends on cannot fork.

codex_steer.go now records why the grace differs there, alongside the note
that every interrupt leaves a dangling tool call in the rollout (codex logs
"Custom tool call output is missing" on the resume and tolerates it).

Assisted-by: Claude
Signed-off-by: Wayne Sun <gsun@redhat.com>
waynesun09 added a commit that referenced this pull request Sep 4, 2026
The stray-process sweep waited a fixed 2s between TERM and KILL. A
maintainer objected on #6753 that this leaves an agent no room to flush
state on SIGTERM, and that objection lands hardest on the codex steer path:
ClearIterationArtifacts sweeps leftovers from a run that is already over,
but codexSteerQueue.interrupt stops a process the runner intends to
CONTINUE, one turn of a thread it is about to resume.

The grace is now a parameter of the snippet (__GRACE_TICKS__ for the poll
loop, __GRACE_LABEL__ for its own comment). ClearIterationArtifacts keeps
2s and renders byte-for-byte what it rendered before —
testdata/kill_stray_processes.sh is unchanged, and its golden test still
passes untouched. The codex interrupt gets 10s through interruptSweep,
which is the default of the existing injectable `sweep:` field, so tests
can still replace the whole sweep.

The exec timeout scales with the grace rather than staying at a flat 15s.
That bound exists to catch a hung gateway; left fixed, raising the grace
would have meant the timeout fired during the TERM wait and the KILL pass
never ran — the sweep would have started leaking exactly the processes it
exists to remove. Default stays 15s (2s + 13s headroom), the interrupt gets
23s, and a test asserts the timeout always outlasts the grace with room to
spare.

The interrupt rendering is pinned in
testdata/kill_stray_processes_interrupt.sh the same way the default one is,
and kill_stray_processes_test.sh now takes an optional snippet path so
either can be executed under a real shell. Both were run: the interrupt
golden passes, taking ~26s against the default's ~16s because the
TERM-ignoring fixture now takes its full 10s before the KILL lands — which
is the evidence that the longer wait actually elapses and the KILL pass
still works, rather than just that the string renders. A test also
normalises the two renderings and asserts they differ ONLY in the grace, so
the process-selection logic the sandbox depends on cannot fork.

codex_steer.go now records why the grace differs there, alongside the note
that every interrupt leaves a dangling tool call in the rollout (codex logs
"Custom tool call output is missing" on the resume and tolerates it).

Assisted-by: Claude
Signed-off-by: Wayne Sun <gsun@redhat.com>
waynesun09 added a commit that referenced this pull request Sep 4, 2026
The stray-process sweep waited a fixed 2s between TERM and KILL. A
maintainer objected on #6753 that this leaves an agent no room to flush
state on SIGTERM, and that objection lands hardest on the codex steer path:
ClearIterationArtifacts sweeps leftovers from a run that is already over,
but codexSteerQueue.interrupt stops a process the runner intends to
CONTINUE, one turn of a thread it is about to resume.

The grace is now a parameter of the snippet (__GRACE_TICKS__ for the poll
loop, __GRACE_LABEL__ for its own comment). ClearIterationArtifacts keeps
2s and renders byte-for-byte what it rendered before —
testdata/kill_stray_processes.sh is unchanged, and its golden test still
passes untouched. The codex interrupt gets 10s through interruptSweep,
which is the default of the existing injectable `sweep:` field, so tests
can still replace the whole sweep.

The exec timeout scales with the grace rather than staying at a flat 15s.
That bound exists to catch a hung gateway; left fixed, raising the grace
would have meant the timeout fired during the TERM wait and the KILL pass
never ran — the sweep would have started leaking exactly the processes it
exists to remove. Default stays 15s (2s + 13s headroom), the interrupt gets
23s, and a test asserts the timeout always outlasts the grace with room to
spare.

The interrupt rendering is pinned in
testdata/kill_stray_processes_interrupt.sh the same way the default one is,
and kill_stray_processes_test.sh now takes an optional snippet path so
either can be executed under a real shell. Both were run: the interrupt
golden passes, taking ~26s against the default's ~16s because the
TERM-ignoring fixture now takes its full 10s before the KILL lands — which
is the evidence that the longer wait actually elapses and the KILL pass
still works, rather than just that the string renders. A test also
normalises the two renderings and asserts they differ ONLY in the grace, so
the process-selection logic the sandbox depends on cannot fork.

codex_steer.go now records why the grace differs there, alongside the note
that every interrupt leaves a dangling tool call in the rollout (codex logs
"Custom tool call output is missing" on the resume and tolerates it).

Assisted-by: Claude
Signed-off-by: Wayne Sun <gsun@redhat.com>
waynesun09 added a commit that referenced this pull request Sep 5, 2026
The stray-process sweep waited a fixed 2s between TERM and KILL. A
maintainer objected on #6753 that this leaves an agent no room to flush
state on SIGTERM, and that objection lands hardest on the codex steer path:
ClearIterationArtifacts sweeps leftovers from a run that is already over,
but codexSteerQueue.interrupt stops a process the runner intends to
CONTINUE, one turn of a thread it is about to resume.

The grace is now a parameter of the snippet (__GRACE_TICKS__ for the poll
loop, __GRACE_LABEL__ for its own comment). ClearIterationArtifacts keeps
2s and renders byte-for-byte what it rendered before —
testdata/kill_stray_processes.sh is unchanged, and its golden test still
passes untouched. The codex interrupt gets 10s through interruptSweep,
which is the default of the existing injectable `sweep:` field, so tests
can still replace the whole sweep.

The exec timeout scales with the grace rather than staying at a flat 15s.
That bound exists to catch a hung gateway; left fixed, raising the grace
would have meant the timeout fired during the TERM wait and the KILL pass
never ran — the sweep would have started leaking exactly the processes it
exists to remove. Default stays 15s (2s + 13s headroom), the interrupt gets
23s, and a test asserts the timeout always outlasts the grace with room to
spare.

The interrupt rendering is pinned in
testdata/kill_stray_processes_interrupt.sh the same way the default one is,
and kill_stray_processes_test.sh now takes an optional snippet path so
either can be executed under a real shell. Both were run: the interrupt
golden passes, taking ~26s against the default's ~16s because the
TERM-ignoring fixture now takes its full 10s before the KILL lands — which
is the evidence that the longer wait actually elapses and the KILL pass
still works, rather than just that the string renders. A test also
normalises the two renderings and asserts they differ ONLY in the grace, so
the process-selection logic the sandbox depends on cannot fork.

codex_steer.go now records why the grace differs there, alongside the note
that every interrupt leaves a dangling tool call in the rollout (codex logs
"Custom tool call output is missing" on the resume and tolerates it).

Assisted-by: Claude
Signed-off-by: Wayne Sun <gsun@redhat.com>
waynesun09 added a commit that referenced this pull request Sep 6, 2026
The stray-process sweep waited a fixed 2s between TERM and KILL. A
maintainer objected on #6753 that this leaves an agent no room to flush
state on SIGTERM, and that objection lands hardest on the codex steer path:
ClearIterationArtifacts sweeps leftovers from a run that is already over,
but codexSteerQueue.interrupt stops a process the runner intends to
CONTINUE, one turn of a thread it is about to resume.

The grace is now a parameter of the snippet (__GRACE_TICKS__ for the poll
loop, __GRACE_LABEL__ for its own comment). ClearIterationArtifacts keeps
2s and renders byte-for-byte what it rendered before —
testdata/kill_stray_processes.sh is unchanged, and its golden test still
passes untouched. The codex interrupt gets 10s through interruptSweep,
which is the default of the existing injectable `sweep:` field, so tests
can still replace the whole sweep.

The exec timeout scales with the grace rather than staying at a flat 15s.
That bound exists to catch a hung gateway; left fixed, raising the grace
would have meant the timeout fired during the TERM wait and the KILL pass
never ran — the sweep would have started leaking exactly the processes it
exists to remove. Default stays 15s (2s + 13s headroom), the interrupt gets
23s, and a test asserts the timeout always outlasts the grace with room to
spare.

The interrupt rendering is pinned in
testdata/kill_stray_processes_interrupt.sh the same way the default one is,
and kill_stray_processes_test.sh now takes an optional snippet path so
either can be executed under a real shell. Both were run: the interrupt
golden passes, taking ~26s against the default's ~16s because the
TERM-ignoring fixture now takes its full 10s before the KILL lands — which
is the evidence that the longer wait actually elapses and the KILL pass
still works, rather than just that the string renders. A test also
normalises the two renderings and asserts they differ ONLY in the grace, so
the process-selection logic the sandbox depends on cannot fork.

codex_steer.go now records why the grace differs there, alongside the note
that every interrupt leaves a dangling tool call in the rollout (codex logs
"Custom tool call output is missing" on the resume and tolerates it).

Assisted-by: Claude
Signed-off-by: Wayne Sun <gsun@redhat.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ready-for-merge All reviewers approved — ready to merge risk/elevated PR risk: elevated

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants