runner: validation retry resumes the agent session instead of re-running it from a blank session

[waynesun09]

What happened

A validation_loop retry re-runs the agent as a fresh process in the same sandbox: ClearIterationArtifacts sweeps stray processes, deletes the iteration's output and the runtime's session store, and iteration 2 starts a new session with the previous validation failure injected into the prompt (#1050, #6494). The agent keeps the working tree but loses everything else it knew: what it tried, why, and what it already ruled out. The process kill between iterations is also what forces the stray-process sweep from #6753 into the validation loop, which #6961 objects to.

#6961 proposes the alternative: do not terminate the agent process; validate when the agent says it is done. #6959 already built the machinery for that shape, for work-item updates: one sandbox, and one session per steerable rt.Run, with the runner talking to the live session at result boundaries.

What #6959 shipped and what it did not:

• Shipped: session-id capture for all three runtimes in RunMetrics.SessionID (Claude system/init session_id, Codex thread.started thread_id, pi session id). Under Steerable, the runner is notified at the runtime's result boundary before deciding to Steer or Settle (steerTurnEndHandler in internal/cli/steer.go; coalescing channel, depth 16): Claude reports per turn, pi per prompt (rpc mode), Codex only at process exit. Claude and pi stay alive on stdin between turns; Codex waits for work after a natural exit and resumes the same thread. Today a steer to an idle Codex queue still fires the interrupt sweep (codexSteerQueue.enqueue checks only that the thread id is known); that is a feat(dispatch)!: steer the agent run already in flight on work-item updates #6959 defect being fixed there, and this issue depends on the fix.
• Not shipped: anything on the validation path. Output extraction, transcript extraction and the validation script still run after rt.Run returns, once per iteration (run.go steps 9b–9e), and ClearIterationArtifacts still deletes pi's --session-dir and Codex's $CODEX_HOME/sessions. For Claude it runs rm -rf <config>/*.jsonl (top level only) while the session lives under <config>/projects/<cwd>/<id>.jsonl, so if Claude stores the session under projects/ (inferred from our clear and extraction code, not yet confirmed from a captured run) it survives by accident and iteration 2's transcript extraction (a recursive find) also picks up iteration 1's file.

Proposed change

Deliver validation feedback as the next turn of the live session instead of killing the agent and starting a new one.

Runner loop. When the harness has validation_loop, run the agent Steerable and, at each turn end: download the output, run the validation script on the host, then either Steer the failure text as the next user message (feedback_mode: append semantics) or Settle on pass. Clear only output/ between turns, never the session store. max_iterations remains the cap on turns.

Per runtime.

• Claude Code and pi: the live session takes the next message on the existing mailbox path; no process exit, no kill.
• Codex: no live channel, so the runner's steer after the natural exit becomes codex exec <flags> resume <thread_id> - on the same thread (0.152.1; flags before resume). Needs the idle-aware queue fix in feat(dispatch)!: steer the agent run already in flight on work-item updates #6959 so the steer does not fire the interrupt sweep.
• Any runtime without Steerer (the type assertion already exists): today's fresh-process retry with the feedback in the prompt, and a line in the run summary saying so.

Interaction with the steer watcher (#6959), so the two paths do not collide.

• Steerable is currently set only when the harness enables steer:; the validation path needs its own switch that also sets it, so a harness with validation_loop and steer off still gets the live retry.
• A validation-feedback SteerMessage carries FollowUpRunID 0 ("not from a run"); it must not enter the consumed-run marker and must not count against steer.max_steers, which budgets follow-up runs. max_iterations is its cap.
• A remaining-time floor like the watcher's MinRemaining is needed here too: today that floor lives only inside the follow-up-run watcher, which is not created when steer: is off, so the validation path must carry its own check. A retry with less than the floor left ends the run instead of starting a turn the exec timeout would cut off.
• Feedback text must never reach the agent through argv (it is the most attacker-influenced string in the run). Today the mailbox seed and the Codex prompt are written with printf '%s' <quoted> inside the sh -c string the runner execs, so the text is in that intermediate shell's argv and in OpenShell's host-side command preview; the agent CLI itself reads stdin. Fixing that (plumb the exec request's stdin field through sandbox.ExecContext, or upload-then-append) is a feat(dispatch)!: steer the agent run already in flight on work-item updates #6959 follow-up this issue inherits. Note the field exists on the gRPC request (ExecSandboxRequest.stdin is a one-shot bytes payload) but the openshell sandbox exec CLI fullsend shells out to does not expose it, so the fix means the gRPC path or the upload path, not a new CLI flag.
• Metrics: per-turn usage as a steered session reports it today.

Transcripts. Iteration 1's session file is attributed to iteration 1 once, not re-extracted on the next turn.

Out of scope: the stray-process sweep itself. With no kill between validation turns it leaves the validation loop; it remains for the Codex mid-turn steer interrupt. NVIDIA/OpenShell#3159 was closed (OpenShell will not add per-exec termination), so any replacement is fullsend-side and is discussed on #6961. Two source facts constrain that replacement (see research/openshell-sandbox-process-model.md in the workspace): an exec'd command is a sibling of the sandbox main process under the PID 1 supervisor, and on the --no-tty path it inherits the supervisor's process group, so a group kill from inside the sandbox is only safe if the launched shell first creates its own session.

Sequencing. The pi launch command builder in internal/runtime/pi_run.go is rewritten by both #6754 (plugins under one key: JITI_* env pinning, -e extension list) and #6959 (--mode rpc --session-id under Steerable), and those two conflict with each other on that file today. This issue lands after both, on top of whichever merges second. #6754 also documents its extension tree-hash and hook-digest guards in terms of "between iterations" (docs/contributing/runtime-implementation.md, and the refusal message in pi_run.go): under a live-session retry pi is not restarted between validation turns, so those per-process guards run once per session rather than once per iteration and the "previous iteration's process rewrites the tree before the next import" window disappears for pi (there is no second import); the wording there needs updating when this lands, and the Codex resume path, which is a new process per turn, keeps the per-process guards as they are. No MCP server, remote or sidecar (#6961 floated one): it costs context per turn, VM resources and a second control surface next to the harness. The agent's early signal is the existing in-sandbox tool fullsend-check-output (#2777 for the code agent); the turn boundary is the contract the runner relies on, and the host-side validation script and post-script keep the schema and outcome checks.

Validation criteria

• A two-iteration code or fix run on each runtime shows the second turn continuing the first session: same RunMetrics.SessionID, the agent's next message references the validation failure text, and the iteration-1 transcript is extracted exactly once.
• No process kill and no stray-process sweep between validation turns on any runtime (the sweep log line is absent); pi and Codex session stores survive, output/ does not.
• With steer.enabled: false and validation_loop set, the live retry still happens; with both on, a validation turn does not consume a follow-up run or decrement max_steers.
• Forcing the fallback (runtime without Steerer, or session file removed) produces the fresh-process retry with the feedback in the prompt and a visible "live retry unavailable" line.
• No change to a single-iteration run.

Related

• Should not have giant hacks to reuse sandbox for validation loop #6961: the objection this answers; the sweep leaves the validation loop.
• Steer the in-flight agent run on work-item updates instead of cancelling it #6957 step 1 and feat(dispatch)!: steer the agent run already in flight on work-item updates #6959: session-id capture, turn-end observation, mailbox delivery; this reuses them.
• Allow developers to resume or continue a fullsend agent session locally #459 (resume a CI session locally) is the second consumer of keeping the session store.
• Implement validation loop feedback so agents can self-correct #1050 and Run pre-commit inside the code/fix validation loop so a lint block becomes a retry, not a dead end #6494 defined today's fresh-process retry with feedback; this keeps their feedback contract and changes only how the retry is delivered.
• ADR 0098 (docs(adr): serialize agent runs and coalesce subsequent events #6909, preserve the active run and coalesce; each run reconciles current state) and ADR 0099 (docs(adr): adopt entity-first harness evaluation #6956, entity-first evaluation with per-harness receipts), both by @ifireball: the validation turn is the active run reconciling before it ends, and the check outcome is the receipt; the design here must not conflict with either.
• fix(runtime): terminate stray sandbox processes between iterations #6753 added the between-iteration sweep; feat(sandbox): let the caller terminate a sandbox exec command and everything it spawned NVIDIA/OpenShell#3159 asks for the primitive that would replace its remaining use.

[fullsend-ai-triage]

🤖 Finished Triage · ✅ Success · Started 2:56 PM UTC · Completed 3:01 PM UTC

Commit: 48eac1a · View workflow run →

Runtime: claude · Model: opus → claude-opus-4-6 · Effort: high · Cost: $0.86

[fullsend-ai-triage]

This issue is blocked on #6959, which provides the session-id capture infrastructure (RunMetrics.SessionID for all three runtimes) that the resume-per-runtime logic depends on. #6959 also needs to land before the keep-session clear mode can be implemented, since the clear-mode change to ClearIterationArtifacts must preserve the session stores that #6959 introduces.

Once #6959 merges, this issue is immediately actionable.

Note: NVIDIA/OpenShell#3159 (sandbox process termination primitive) is explicitly out of scope per the issue body — the existing stray-process sweep from #6753 stays regardless of whether retry resumes or restarts.

Blocked by:

• feat(dispatch)!: steer the agent run already in flight on work-item updates #6959

---

Labels: Adding blocked label since #6959 must merge before this work can proceed; adding priority/high to match parent issue #6957 priority.

[fullsend-ai-triage]

🤖 Finished Triage · ✅ Success · Started 3:36 PM UTC · Completed 3:40 PM UTC

Commit: 48eac1a · View workflow run →

Runtime: claude · Model: opus → claude-opus-4-6 · Effort: high · Cost: $0.61

[fullsend-ai-triage]

Re-checked the prerequisite: PR 6959 is still open and unmerged. That PR ships the session-id capture (RunMetrics.SessionID for Claude, Codex, and pi), turn-end observation (steerTurnEndHandler), and the Steerable runner mode that this issue reuses for live validation feedback. Once 6959 lands, this issue becomes immediately actionable.

No open PR addresses this issue yet, and no duplicates were found (6961 is the motivating objection, not a duplicate).

Blocked by:

• feat(dispatch)!: steer the agent run already in flight on work-item updates #6959