Skip to content

feat(harness): lease a pool sandbox for prompt leaves - #201

Merged
pdettori merged 1 commit into
rossoctl:mainfrom
pdettori:feat/prompt-leaf-pool-aware
Sep 1, 2026
Merged

feat(harness): lease a pool sandbox for prompt leaves#201
pdettori merged 1 commit into
rossoctl:mainfrom
pdettori:feat/prompt-leaf-pool-aware

Conversation

@pdettori

@pdettori pdettori commented Sep 1, 2026

Copy link
Copy Markdown
Member

Summary

A kind:"prompt" leaf resolved its own sandbox from the environment — ADR-0028 deferred the lease with "prompt leaves inherit /turn's sandbox routing" — so it never consulted one. Two consequences:

  1. SH_REMOTE_SANDBOX / SH_RELAY_ADDR are read only on the lease path, so a remote sandbox was unreachable from a prompt leaf.
  2. Worse, on any deployment configured the way deploy/knative/service.yaml is (a KAGENTI_SANDBOX_POOL_SELECTOR, no KAGENTI_SANDBOX_POD), resolveSandboxConfig returned null and the leaf's tool calls ran in the harness container's own filesystem — silently, and indistinguishably from a sandbox that answered. Since that container is node:22-alpine, a prompt reading /etc/os-release would confidently report Alpine having touched no sandbox at all.

The driver is the remote-sandbox demo, which needs a free-form prompt to execute its tool calls on a laptop container reached over the sandbox relay.

Changes

Harness

  • runPromptLeaf leases through selectPoolSandbox exactly as the converge and solve paths do — heartbeat while the turn runs, release and close the transport in a finally.
  • executeTurn gains an optional pre-leased sandbox. resolveTurnSandbox returns it verbatim when present and otherwise falls back to resolveSandboxConfig, so /turn is unchanged. It returns exactly k8sSandboxExtension's argument and is passed through untransformed, so a leased transport cannot be dropped by a field-by-field rebuild.
  • This is a superset, not a swap: with no pool selector configured, selectPoolSandbox performs that same single-pod resolution itself and returns null when nothing is set.

Two contract changes, both recorded in the ADR amendment:

  • Saturation propagates as {status:"failed", reason:"saturated"} — the sync /runs path already bounded-waits then 503s on it, and classifyOutcome already keeps it retryable for the async queue. The alternative ADR-0028 rejected ("imports the saturation/503 path") is accepted as a consequence rather than avoided; suppressing it would have taken extra code to be less consistent.
  • A throw from the turn returns failed/error instead of escaping as a 500, matching converge and solve.

Demo — driven by a free-form prompt, since the reply names the OS it read rather than handing over a flag to trust:

  • lib-relay.sh gains dispatch_prompt and assert_reply_contains/_lacks alongside the verdict pair, which relay-leaf-smoke.sh keeps — its binary CLEAR/FLAGGED is the stronger gate for a live smoke that cannot run in CI. tests/lib-relay-shared.test.sh locks both pairs and records why they diverge.
  • demo-remote-worker.sh gains Act 3, which it previously stopped short of while the doc claimed every step was asserted: the planted marker (fail-closed if the pod unexpectedly has the file) and the presence-gone teardown, skipped under --keep since that flag promises the worker stays running.

Testing

  • 10 new unit tests, each watched fail first: 4 for resolveTurnSandbox, 6 for the lease (transport handed to the turn, remoteSandbox forwarded, request-scoped pool selector, saturation mapped, lease released on success and on a throw).
  • make test — 686 pass, 0 fail (includes the deploy shell tests and Go tests). make typecheck clean. shellcheck -x clean on all three scripts apart from SC2329/SC2015 informational findings that predate this change.
  • Live end-to-end on kind (make demo-remote-sandbox DEMO_ARGS=--reuse-cluster): 12 assertions passed, 0 failed.
backend                            | what the model said it read
-----------------------------------+----------------------------
in-cluster pod (sandbox-0)         | The system is running Alpine Linux version 3.20.10.
remote host container              | The OS is Red Hat Enterprise Linux version 9.8 (Plow).

PASS: pool selector '...pool=demo-remote-only' matches 0 Running pods
PASS: marker exists only in sh-demo-remote-worker on this host; sandbox-0 has no /tmp/proof.txt
PASS: C/remote/marker: reply names 'tuscan-lentils-16362-43081'
PASS: presence record for sbx-laptop-demo cleared when the worker's Attach stream closed

Same prompt on both backends. B/remote: reply names 'Red Hat' is the assertion that fails on the old code, where both runs report the harness container's Alpine.

Not in scope

A workload-addressed prompt leaf (workloadId) still ignores the workload's own sandboxSelector, and the API boundary still logs that warning. The envelope's sandboxPoolSelector is honoured now, so only the workload-resolver special case remains — whether a workload's pool should bound its prompt leaves is a separate decision, noted as still deferred in the ADR.

A `kind:"prompt"` leaf resolved its own sandbox from the environment
(ADR 0028: "prompt leaves inherit /turn's sandbox routing"), so it never
consulted a lease. Two consequences: SH_REMOTE_SANDBOX/SH_RELAY_ADDR are
read only on the lease path, making a remote sandbox unreachable from a
prompt leaf; and on a pool-selector-only deployment like service.yaml's,
resolveSandboxConfig returned null and the leaf's tool calls ran in the
harness container itself -- silently, and indistinguishably from a
sandbox that answered.

runPromptLeaf now leases through selectPoolSandbox as the converge and
solve paths do (heartbeat while the turn runs, release in a finally) and
hands the leased sandbox to the turn. executeTurn gains an optional
pre-leased `sandbox`; resolveTurnSandbox returns it verbatim when present
and otherwise falls back to resolveSandboxConfig, so /turn is unchanged.
This is a superset rather than a swap: with no pool selector configured,
selectPoolSandbox performs that same single-pod resolution itself.

Two contract changes, both recorded in the ADR amendment: saturation
propagates as failed/saturated (the sync route bounded-waits then 503s;
classifyOutcome already keeps it retryable for the async queue), and a
throw from the turn returns failed/error instead of escaping as a 500 --
matching converge and solve.

Drive the remote-sandbox demo with a free-form prompt, since the reply
names the OS it read rather than handing over a flag to trust.
lib-relay.sh gains dispatch_prompt and assert_reply_contains/_lacks
alongside the verdict pair, which relay-leaf-smoke.sh keeps: its binary
CLEAR/FLAGGED is the stronger gate for a live smoke that cannot run in
CI. lib-relay-shared.test.sh locks both pairs and records why they
diverge. The demo script also gains Act 3, which it previously stopped
short of while the doc claimed every step was asserted: the planted
marker (fail-closed if the pod unexpectedly has the file) and the
presence-gone teardown, skipped under --keep since that flag promises the
worker stays running.

A workload-addressed prompt leaf still ignores the workload's own
sandboxSelector; that remains deferred and is noted in the ADR.

Assisted-By: Claude (Anthropic AI) <noreply@anthropic.com>
Signed-off-by: Paolo Dettori <dettori@us.ibm.com>
@pdettori
pdettori merged commit d2a9167 into rossoctl:main Sep 1, 2026
11 checks passed
@pdettori
pdettori deleted the feat/prompt-leaf-pool-aware branch September 1, 2026 19:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant