Demo: one-command laptop showcase — harness on kind, remote worker as a docker container

[pdettori]

Goal

One command on a laptop that demonstrates the SandboxTransport headline — a sandbox
outside the cluster, with zero inbound rules, executing a leaf's tool calls — and that
someone can run without a cluster account, a VPN to OpenShift, or any hand-patching.

Today that story is only reachable by someone willing to deploy to a cluster and edit
several manifests by hand. I did exactly that to validate it (see the walkthrough below);
it works, but it took a dozen steps and two traps that are not written down anywhere.

Shape

laptop
├─ kind cluster:  Knative + Redis + harness (ksvc) + sandbox-relay
└─ docker run:    remote-worker  ──dials out──▶  relay

The worker runs as a plain docker run on the host, not as a pod. That is the whole
point: it is outside the cluster, it has no inbound ports, and it reaches the relay by
dialing out. A worker deployed as a pod (which is what worker-example.yaml does)
demonstrates the plumbing but not the driver.

The proof it must show

Verdict text that could only come from the container's own filesystem. The trick that
worked well when I validated this: the in-cluster sandbox-N pods are Alpine and the
worker image is RHEL, so a leaf grepping /etc/os-release gives opposite answers
depending on which backend ran:

Backend	pattern Alpine	model's stated reason
kubectl sandbox pod	FLAGGED	"…running Alpine Linux"
remote worker	CLEAR	"…this system is running Red Hat Enterprise Linux 9.8"

An A/B in one script — same request, opposite verdicts — is far more convincing than a
single green run, because the model names the OS it actually read.

Features worth surfacing (each is already implemented and tested)

• Inverted connectivity — docker run with no -p, no inbound rules at all.
• Live streaming — a long bash whose chunks arrive as produced, not batched.
• Abort mid-stream — kill a runaway; the process group dies (a backgrounded
  grandchild stops too, which is the part that is easy to get wrong).
• Dual-ended timeout — timeout_s enforced worker-side, timeout:<n> surfaced.
• Reconnect → dedup — docker stop the worker mid-flight, restart it, watch it
  reattach and not re-run the redelivered exec.
• Presence mirror — redis-cli HGETALL sh:sandbox:records before/after, showing
  registration is the live stream.
• Trust model — docker inspect the worker: a bearer token and nothing else. No LLM
  key, no kubeconfig, no orchestration.
• Transport swappability — flip SH_REMOTE_SANDBOX and rerun; identical contract,
  different machine.

Two traps the script must handle (both cost me a failed run)

1. Enabling SH_REMOTE_SANDBOX=1 does not route to the worker. select-sandbox
builds candidates = [...pods, ...grpcRecs] and picks least-loaded-first, so with idle
in-cluster sandbox pods present a pod wins and the demo silently proves nothing — my
first remote run returned "Alpine" and looked like a success. This is correct behavior
(the remote worker is a pool peer, not a replacement), so the demo must either point
KAGENTI_SANDBOX_POOL_SELECTOR at a label matching no pods, or stand up a cluster with no
local sandboxes. Worth stating loudly in the README too.

2. deploy/knative/relay-deployment.yaml omits SH_RELAY_TOKEN. Relay auth is
fail-closed, so applying that manifest as-is yields a relay that rejects every Attach.
Either add the env to the manifest or have the script set it. (Small enough to fix
independently of this issue.)

Also note kubectl set env does not work on a Knative Service ("no kind Service is
registered") — the script needs a merge patch, and the patch must preserve the container's
volumeMounts or Knative's webhook rejects it.

Networking

The worker is on the host and the relay is in kind, so the script needs one of:

• kubectl port-forward svc/sandbox-relay 8443:8443 and run the worker with
  RELAY_ADDR=host.docker.internal:8443 — simplest, works on Docker Desktop and Podman.
• attach the worker container to kind's docker network and address the node directly.

Port-forward is probably the right default: it is the same mechanism remote-worker/DESIGN.md
already documents for the laptop-worker flow, and it makes the "dials out through a tunnel"
story explicit.

Acceptance

• make demo-remote-sandbox (or deploy/kind/demo-remote-worker.sh) goes from no
  cluster to a passing A/B in one invocation, and prints what it is proving at each step.
• Teardown is one command and leaves no kind cluster, container, or loaded image behind.
• The A/B assertion fails loudly if the exec landed on a pod instead of the worker —
  it must not be possible to get a green demo from the wrong backend (trap 1).
• README section with the architecture sketch, the one-liner, and the trust-model note.
• Works with the worker image built locally; no registry push required.

Prerequisites and honest limits

• Needs an LLM reachable from the cluster for the leaf's model call. In my validation that
  was an internal LiteLLM endpoint; the demo should fail with a clear message when the
  configured model is unreachable rather than timing out mysteriously.
• Docker/Podman + kind + kubectl + jq.
• remote-worker/build-image.sh currently targets an OpenShift internal-registry build;
  the demo needs a plain local build plus kind load docker-image (I did this by hand with
  a 6-line Dockerfile over a cross-compiled linux/arm64 binary — worth folding into the
  script, and note the binary arch must match the kind node).

Validated end-to-end already (2026-08-27)

For the avoidance of doubt, the path this issue automates has been run manually and works:
worker pod attached to an in-cluster relay via ClusterIP DNS, presence mirrored into Redis,
harness on SH_REMOTE_SANDBOX=1, and a leaf returning CLEAR for "Alpine" with the reason
"this system is running Red Hat Enterprise Linux 9.8" — i.e. the leaf's file tools ran on
the remote worker, with no kubectl exec anywhere in the path. The gated relay-interop test
(SH_LIVE_RELAY=1) also passes against the real TypeScript relay, and the real
GrpcRelayTransport has been driven against the real relay and real worker.

What this issue adds is not new capability — it is making that reproducible in one command
on a laptop, which is what turns it into a demo.

Part of the SandboxTransport epic #89; complements the live gate in #88 (which covers
Kind → OCP conformance rather than a laptop showcase). Worker follow-ups: #173.

Assisted-By: Claude Code

[pdettori]

Research findings — three assumptions in the issue have moved

Before planning, I traced the current state of main. Three things the issue describes are
no longer accurate, and they shrink the work considerably.

1. Trap 2 is already fixed. deploy/knative/relay-deployment.yaml:45 sets
SH_RELAY_TOKEN: dev-token, with a comment explaining fail-closed auth and flagging that the
same file is included unpatched by the OCP overlay (so it is a live credential there —
secretKeyRef, #173, is still the real fix). Nothing for this issue to do.

2. relay-leaf-smoke.sh already implements most of the proof. The gated live smoke from #88
already does the Alpine/RHEL A/B, verifies the discriminator holds before relying on it
(refusing to assert otherwise), defends trap 1 with a non-matching pool selector, snapshots the
ksvc env and restores it from an EXIT trap, asserts Redis presence with transport=grpc, and
runs a control pass on the pod path afterwards.

So the delta for this issue is not the A/B. It is:

• the worker as a host container rather than an in-cluster pod (the actual headline),
• zero-to-cluster orchestration in one invocation,
• demo narration instead of PASS/FAIL smoke output,
• the nine feature vignettes.

3. No hand-rolled Dockerfile is needed. remote-worker/Dockerfile already builds
self-contained from the repo root (golang:1.25-alpine → ubi9/ubi-minimal, with bash +
coreutils + file). One docker build at host arch. And because the worker runs as a host
container, there is no kind load and no arch-matching against the kind node at all — that
caveat only ever applied to the pod path. The RHEL base doubles as the discriminator for free.

One new trap, resolved empirically

The issue proposes port-forward + RELAY_ADDR=host.docker.internal:8443. kubectl port-forward
binds 127.0.0.1 only, so whether a container can reach it depends on the runtime:

Runtime	host.docker.internal →	Default port-forward bind reachable?
Podman machine / Docker Desktop (macOS)	gvproxy host gateway, forwards to host loopback	yes (measured)
Native Linux Docker (--add-host=...:host-gateway)	docker0 bridge IP, not loopback	no — needs --address 0.0.0.0

Measured on Podman 5.7.1 behind the docker CLI: host.docker.internal resolves to
192.168.127.254 and the default bind is reachable, with a no-listener control confirming the
probe is real. So the common laptop case needs no 0.0.0.0 bind and no LAN exposure. The script
will probe reachability from a throwaway container and only widen the bind if the probe fails,
printing which path it took — portable without exposing the relay by default.

Plan

Landing this in two PRs. PR 1 closes every acceptance box; PR 2 adds the showcase vignettes,
which are the bulk of the line count and none of the acceptance criteria.

flowchart TD
    A[preflight: docker/kind/kubectl/jq + model creds] --> B{cluster?}
    B -->|--reuse-cluster and healthy| D
    B -->|else| C[kind create + setup-kind.sh]
    C --> D[build worker image at host arch<br/>no kind load: it runs on the host]
    D --> E[apply relay-deployment.yaml]
    E --> F[port-forward 8443 + probe from a container<br/>widen bind only if probe fails]
    F --> G["docker run -d --name sh-demo-worker<br/>NO -p flags — dials out"]
    G --> H[assert Redis presence transport=grpc]
    H --> I[validate discriminator:<br/>pod=Alpine vs worker=RHEL]
    I --> J["run A — pod path: 'Alpine' must be FLAGGED"]
    J --> K[flip ksvc: SH_REMOTE_SANDBOX=1<br/>+ selector matching ZERO pods]
    K --> L["run B — remote: 'Alpine' CLEAR and 'Red Hat' FLAGGED"]
    L --> M[restore env via EXIT trap]
    M --> N[print A/B table with the model's own stated reason]

Loading

PR 1 — the acceptance criteria

• deploy/knative/demo-remote-worker.sh (matching the existing convention; deploy/kind/
  does not exist and its one file would depend on deploy/knative/lib.sh anyway).
• deploy/knative/lib-relay.sh — extract the worker-agnostic A/B logic so the demo and the
  ST5: integration & live gate — cross-transport conformance + end-to-end leaf (Kind → OCP) #88 gate cannot drift: dispatch_pattern, assert_verdict, validate_discriminator,
  the env snapshot/flip/restore, and the presence assertion. validate_discriminator takes the
  two os-release strings rather than fetching them, since the gate reads the worker via
  kubectl exec and the demo via docker exec — shared assertion, local fetch.
• relay-leaf-smoke.sh re-pointed at the shared lib, behaviour unchanged.
• Makefile: demo-remote-sandbox and demo-remote-sandbox-teardown.
• README section in README-worker.md: architecture sketch, the one-liner, the trust-model
  note, and trap 1 stated loudly.

How trap 1 is made structurally impossible, rather than merely detected: before dispatching
the remote runs, the script asserts the configured pool selector matches zero Running pods.
select-sandbox.ts:103 builds candidates = [...pods, ...grpcRecs], so with zero pods the worker
is the only candidate and no lease ordering can route around it. The double-pattern assertion
(Alpine CLEAR and Red Hat FLAGGED) then catches a pod-landed exec either way, and the
summary prints the model's own stated reason so the OS it actually read is visible, not inferred.

Teardown removes the container, the relay, the kind cluster and the built image.

PR 2 — the vignettes (deferred)

Live streaming · abort mid-stream (process group death) · dual-ended timeout · reconnect→dedup ·
presence mirror before/after · docker inspect trust model · transport swappability.

Honest limits

Unchanged from the issue: needs an LLM reachable from the cluster. The script fails with an
explicit message when the model is unreachable rather than timing out mysteriously — the existing
assert_verdict already distinguishes "empty response, likely model unreachable" from a wrong
verdict, and that behaviour comes along with the extraction.

Assisted-By: Claude Code

[pdettori]

Closing: PR #195 satisfies all five acceptance criteria. Verified against the merged commit
fc757a7 rather than the PR branch.

#	Criterion	Result
1	one-command A/B, narrating each step	met — make demo-remote-sandbox, 12 narrated steps
2	one-command teardown, nothing left behind	met, with one deliberate deviation — see below
3	fails loudly if the exec landed on a pod	exceeded — made structurally impossible, not merely detected
4	README: sketch, one-liner, trust-model note	met — README-worker.md:289–376
5	locally-built image, no registry push	met — docker build --load, never kind loaded

The cold-start path is now verified. PR #195 flagged it as its one unvalidated path
("worth one clean run before merge"). That run has now happened, from scratch with no cluster
present, on fc757a7:

PASS: harness answers on http://localhost:8080
PASS: relay answers through the tunnel from the host
PASS: worker sbx-laptop-demo present in sh:sandbox:records with transport=grpc
PASS: discriminator holds: sandbox pod (sandbox-0)=Alpine, worker (host container)=Red Hat
PASS: A/pod-path/Alpine: verdict=FLAGGED (expected FLAGGED)
PASS: pool selector 'sh.kagenti.io/sandbox-pool=demo-remote-only' matches 0 Running pods
PASS: B/remote/Alpine: verdict=CLEAR (expected CLEAR)
PASS: B/remote/RedHat: verdict=FLAGGED (expected FLAGGED)
=== Results: 8 passed, 0 failed ===

setup-kind.sh created the cluster (Creating kind cluster: sh-knative), so the
CREATED_CLUSTER=1 branch and its exit hint were both exercised. The model's own stated
reasons name the OS it actually read — "confirming the system is running Alpine Linux" for the
pod, "the system is Red Hat Enterprise Linux 9.8" for the remote container — which is the
proof this issue asked for rather than a bare green check. Afterwards: no worker container, no
relay, no leaked port-forward, and the harness env restored to its pre-flip
KAGENTI_SANDBOX_POOL_SELECTOR exactly.

Criterion 2, honestly: teardown is one command, but it asks before deleting the kind
cluster and keeps it when there is no terminal to ask on. That is a deliberate deviation
from "leaves no kind cluster behind," and the right one: --reuse-cluster exists so the demo
can run against a long-lived dev cluster, and a fresh --teardown process cannot tell which
kind of cluster it is looking at. Deleting unconditionally would make a cleanup command
destructive to something the demo never owned. It is unit-tested in both directions
(tests/demo-teardown-scope.test.sh, cases 1 and 2) and documented in the README. Recording
it here so a later reader does not mistake the prompt for an oversight.

Two of this issue's stated assumptions had already moved by the time #195 was written, and I
confirmed both: relay-deployment.yaml already set SH_RELAY_TOKEN at the merge base, so
trap 2 was fixed on main; and remote-worker/Dockerfile builds the binary in a
golang:1.25-alpine builder stage from the repo root, so the hand-rolled Dockerfile and local
Go toolchain this issue anticipated are both unnecessary.

Deferred, now tracked in #198: this issue's "Features worth surfacing" list. Four of the
eight bullets are already delivered by the demo (inverted connectivity, transport
swappability, and half each of the presence mirror and trust model), so #198 lists only the
four genuine gaps — live streaming, abort mid-stream, reconnect→dedup, dual-ended timeout —
plus two one-line fixes found while auditing: the demo asserts a presence record appears but
never that it vanishes (assert_presence_gone exists and the gate calls it; the demo does
not), and the docker inspect trust-model check is documented but never executed.

Assisted-By: Claude Code