diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index ddc79fb..559aa94 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -68,6 +68,19 @@ jobs: - name: Test run: pnpm -r test + # The deploy/ shell tests are cluster-free (kubectl/kind/docker are mocked on PATH), so + # they need no node, no pnpm and no cluster -- hence their own fast job rather than a step + # tacked onto `check`. Without this job nothing runs them: security-scans.yml only + # shellchecks deploy/, and the live gate (RELAY_LIVE_SMOKE=1) cannot run in CI at all. + deploy-scripts: + runs-on: ubuntu-latest + timeout-minutes: 5 + steps: + - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 + + - name: Run deploy shell tests + run: make test-deploy + proto: runs-on: ubuntu-latest timeout-minutes: 10 diff --git a/Makefile b/Makefile index 87b9352..34c3388 100644 --- a/Makefile +++ b/Makefile @@ -1,4 +1,4 @@ -.PHONY: lint fmt test typecheck +.PHONY: lint fmt test test-deploy typecheck demo-remote-sandbox demo-remote-sandbox-teardown lint: pre-commit run --all-files @@ -9,9 +9,24 @@ fmt: test: pnpm -r test cd remote-worker && go test ./... + $(MAKE) test-deploy + +# Cluster-free unit tests for the deploy/ shell scripts: kubectl, kind and docker are +# mocked on PATH and only the call log is asserted. Run in CI by the `deploy-scripts` job. +# `set -e` so one failing test file fails the target instead of being scrolled past. +test-deploy: + @set -e; for t in deploy/knative/tests/*.test.sh; do echo "== $$t"; bash "$$t"; done typecheck: cd harness && pnpm exec tsc --noEmit cd packages/k8s-sandbox && pnpm exec tsc --noEmit cd packages/knative-server && pnpm exec tsc --noEmit cd experiments && pnpm exec tsc --noEmit + +# Laptop showcase: harness on kind, remote worker as a host container dialing out. +# See deploy/knative/README-worker.md. Add --reuse-cluster to skip setup on a warm cluster. +demo-remote-sandbox: + bash deploy/knative/demo-remote-worker.sh $(DEMO_ARGS) + +demo-remote-sandbox-teardown: + bash deploy/knative/demo-remote-worker.sh --teardown diff --git a/deploy/knative/README-worker.md b/deploy/knative/README-worker.md index d8cb1a5..285f25e 100644 --- a/deploy/knative/README-worker.md +++ b/deploy/knative/README-worker.md @@ -286,6 +286,94 @@ expects: - **stdin** — feed `exec.stdin` bytes to the child's stdin. `Heartbeat` frames are liveness-only; the harness owns lease counts. +## Laptop demo: worker as a host container (one command) + +Everything above runs the worker as a **pod**. That demonstrates the plumbing but not the +driver: the headline claim is a sandbox *outside* the cluster, with **zero inbound rules**, +executing a leaf's tool calls. One command shows that on a laptop: + +```bash +make demo-remote-sandbox # no cluster -> passing A/B +make demo-remote-sandbox DEMO_ARGS=--reuse-cluster # skip setup on a warm cluster +make demo-remote-sandbox-teardown # remove the container, relay and image +``` + +Teardown removes everything the demo creates, but **asks before deleting the kind cluster**: +`--reuse-cluster` exists so the demo can run against a long-lived dev cluster, and a fresh +`--teardown` process cannot know which kind it is looking at. Answer `y`, or pass +`--yes` to skip the prompt (`DEMO_ARGS=--yes`). With no terminal to ask on, the cluster is +kept. A run that *did* create the cluster says so on exit and points at `--teardown`; a run +against a pre-existing cluster does not. + +``` +laptop +|- kind cluster: Knative + Redis + harness (ksvc) + sandbox-relay +| ^ ^ +| | harness -> relay | worker -> relay +| | sandbox-relay.default.svc:8443 | host.docker.internal:8443 +| | (in-cluster DNS) | (outbound, through a port-forward) +\- docker run: remote-worker --------- dials out ------/ +``` + +Neither address is inbound to the laptop. The worker publishes no ports — `docker run` with +no `-p` at all — and reaches the relay only by dialing *out* through +`kubectl port-forward`. The demo proves the container can reach the tunnel before it starts +the worker, and adapts the bind (`--add-host`, then `--address 0.0.0.0`) for runtimes where +`host.docker.internal` maps to a bridge IP rather than host loopback. + +### Why a green run is not enough + +`SH_REMOTE_SANDBOX=1` alone does **not** route to the worker — see the pool-selection trap +at the top of this file. The demo defends against it twice: + +1. **Structurally.** Before the remote runs it asserts `KAGENTI_SANDBOX_POOL_SELECTOR` + matches **zero Running pods**, so `select-sandbox`'s candidate list contains nothing but + the worker. A pod cannot win a lease it is not a candidate for. +2. **By fingerprint.** The in-cluster pool runs Alpine and the worker image runs RHEL, so a + leaf grepping `/etc/os-release` flips its verdict with the backend — and both directions + are asserted, so an exec that landed on a pod fails one check or the other: + + | backend | pattern `Alpine` | pattern `Red Hat` | model's stated reason | + |---|---|---|---| + | in-cluster sandbox pod | `FLAGGED` | `CLEAR` | "…running Alpine Linux" | + | remote host container | `CLEAR` | `FLAGGED` | "…Red Hat Enterprise Linux 9.8" | + +The discriminator itself is verified before anything relies on it, and the summary prints +the model's own stated reason — so you see the OS it actually read, rather than inferring it +from a green check. + +### Trust model + +Inspect what the worker was given: + +```bash +docker inspect sh-demo-remote-worker --format '{{range .Config.Env}}{{println .}}{{end}}' +``` + +A bearer token, a sandbox id, and a relay address. **No LLM key, no kubeconfig, no +orchestration.** The token must equal the relay's `SH_RELAY_TOKEN`; auth is fail-closed, so +a mismatch rejects the Attach before the stream is ever parked. + +The demo generates that token **fresh per run** and patches it onto the relay, rather than +reusing `relay-deployment.yaml`'s `dev-token`. That value is a repo constant, and therefore +public — which matters because on native Linux Docker the demo may bind the relay port to +`0.0.0.0` (see above), and a LAN-reachable port guarded by a credential anyone can read from +the repo would let a network peer Attach as a sandbox and receive the leaf's exec payloads. +So what `docker inspect` shows is a credential scoped to this one run. Set `SANDBOX_TOKEN` to +pin a value instead. A later `kubectl apply -f relay-deployment.yaml` restores the declared +dev value, so nothing is left patched for other callers. + +### Requirements and limits + +- `docker` (or `podman`) + `kind` + `kubectl` + `jq`. **No local Go toolchain** — + `remote-worker/Dockerfile` builds the binary in a builder stage. The image is built for + the *host*, never `kind load`ed, so its architecture need not match the kind node. +- A model the **cluster** can reach (`ANTHROPIC_API_KEY`, or `ANTHROPIC_AUTH_TOKEN` + + `ANTHROPIC_BASE_URL`). The leaf's verdict is a real model call; the demo fails with an + explicit "model endpoint unreachable" message rather than timing out mysteriously. +- The harness ksvc env is snapshotted and restored from an `EXIT` trap, so an interrupted + run never leaves the cluster pointed at a selector matching nothing. + ## Running the worker outside the cluster The steps above assume the worker runs as a pod in the same cluster, dialing the diff --git a/deploy/knative/demo-remote-worker.sh b/deploy/knative/demo-remote-worker.sh new file mode 100755 index 0000000..b543c4b --- /dev/null +++ b/deploy/knative/demo-remote-worker.sh @@ -0,0 +1,501 @@ +#!/usr/bin/env bash +# deploy/knative/demo-remote-worker.sh +# Laptop showcase: a sandbox OUTSIDE the cluster, with zero inbound rules, executing a +# leaf's tool calls. +# +# laptop +# |- kind cluster: Knative + Redis + harness (ksvc) + sandbox-relay +# \- docker run: remote-worker --dials out--> relay +# +# The worker runs as a plain `docker run` on the host, NOT as a pod. That is the whole +# point: it is outside the cluster, it publishes no ports, and it reaches the relay by +# dialing *out* through a port-forward. A worker deployed as a pod (worker-example.yaml, +# and the relay-leaf-smoke.sh gate) demonstrates the plumbing but not the driver. +# +# What it proves, and why a single green run would not: the harness's select-sandbox.ts +# builds candidates = [...pods, ...grpcRecs] and leases least-loaded-first, so merely +# enabling SH_REMOTE_SANDBOX=1 does NOT route to the worker -- an idle in-cluster sandbox +# pod can still win, and the demo would "pass" having proven nothing. Two defenses: +# +# 1. Before the remote runs, assert the pool selector matches ZERO Running pods, so the +# worker is the only candidate there is (structural, not merely detected). +# 2. Fingerprint the OS the exec actually ran on. The in-cluster pool is Alpine, the +# worker image is RHEL, so a leaf grepping /etc/os-release for "Alpine" flips verdict +# with the backend -- and the model names the OS it read in its own reason. +# +# | backend | pattern "Alpine" | model's stated reason | +# |--------------------|------------------|------------------------------| +# | in-cluster pod | FLAGGED | "...running Alpine Linux" | +# | remote host worker | CLEAR | "...Red Hat Enterprise Linux"| +# +# Usage: +# ./demo-remote-worker.sh # no cluster -> passing A/B, one invocation +# ./demo-remote-worker.sh --reuse-cluster # skip setup if the cluster is already healthy +# ./demo-remote-worker.sh --keep # leave relay + worker running afterwards +# ./demo-remote-worker.sh --teardown # remove container, relay, image; ASK about the cluster +# ./demo-remote-worker.sh --teardown --yes # ...and delete the cluster without asking +# +# Requires: docker (or podman), kind, kubectl, jq -- and credentials for a model the +# cluster can reach (ANTHROPIC_API_KEY, or ANTHROPIC_AUTH_TOKEN + ANTHROPIC_BASE_URL). +# No local Go toolchain: remote-worker/Dockerfile builds the binary in a builder stage. +set -euo pipefail +# Resolve this script's own path BEFORE the cd below. --help reads the header block out of +# the file, and after the cd a relative "$0" no longer resolves -- which is how the demo is +# actually invoked (`make demo-remote-sandbox` runs `bash deploy/knative/demo-remote-worker.sh`). +# The user-facing hints below keep using "$0": those are copy-pasted from the caller's own +# cwd, which is the one "$0" was relative to. +SELF="$(cd "$(dirname "$0")" && pwd)/$(basename "$0")" +cd "$(dirname "$0")" +source ./lib.sh # NS, KSVC, BASE, CURL_OPTS, CURL_HDR, ok/ko, PASS/FAIL, ensure_port_forward +# shellcheck source=./lib-relay.sh +source ./lib-relay.sh # MODEL, claim/abort, dispatch_pattern, assert_verdict, + # validate_discriminator, assert_presence, assert_no_pods_match, + # snapshot/flip/restore_harness_env. Shared with relay-leaf-smoke.sh. + +REPO_ROOT="$(cd ../.. && pwd)" +CLUSTER_NAME="${CLUSTER_NAME:-sh-knative}" +WORKER_IMAGE="${WORKER_IMAGE:-dev.local/remote-worker:demo}" +WORKER_CTR="${WORKER_CTR:-sh-demo-remote-worker}" +RELAY_PORT="${RELAY_PORT:-8443}" +SANDBOX_ID="${SANDBOX_ID:-sbx-laptop-demo}" +# Must equal the relay's SH_RELAY_TOKEN. Auth is fail-closed: a mismatch rejects the Attach +# before the stream is ever parked. Left EMPTY here on purpose -- a per-run random token is +# generated in step 4 and patched onto the relay, rather than reusing relay-deployment.yaml's +# hardcoded `dev-token`. That value is a repo constant and therefore public, which matters +# because step 5 may bind the relay port to 0.0.0.0 on native Linux Docker: a LAN-reachable +# port plus a well-known credential would let anyone on the network Attach as a sandbox and +# receive the leaf's exec payloads. Set SANDBOX_TOKEN to pin a value instead. +RELAY_TOKEN="${SANDBOX_TOKEN:-}" +# The harness reaches the relay by in-cluster DNS; the worker reaches it through the +# tunnel. That asymmetry IS the inverted-connectivity story -- neither address is inbound +# to the laptop. +IN_CLUSTER_RELAY_ADDR="sandbox-relay.${NS}.svc:8443" +# A label no pod carries, so the remote worker is the only lease candidate (defense 1). +REMOTE_ONLY_SELECTOR="sh.kagenti.io/sandbox-pool=demo-remote-only" +PF_LOG="${TMPDIR:-/tmp}/sh-demo-relay-pf.$$.log" + +REUSE_CLUSTER=0 +KEEP=0 +TEARDOWN_ONLY=0 +ASSUME_YES=0 +# Raised only on the path that actually creates the cluster (step 2). --reuse-cluster exists +# so the demo can run against a long-lived dev cluster, so "the cluster exists" must never be +# read as "this run may delete it". +CREATED_CLUSTER=0 +PF_PID="" +KOURIER_PF_PID="" +WORKER_DOCKER_ARGS=() +PF_ADDRESS_ARGS=() + +while [ $# -gt 0 ]; do + case "$1" in + --reuse-cluster) REUSE_CLUSTER=1; shift ;; + --keep) KEEP=1; shift ;; + --teardown) TEARDOWN_ONLY=1; shift ;; + -y|--yes) ASSUME_YES=1; shift ;; + # Print the header block only: from line 2 (skipping the shebang) to the first + # non-comment line. `grep '^#'` would also emit every column-0 comment further down -- + # section headers and function docs -- which is internal commentary, not help. + -h|--help) sed -n '2,/^[^#]/{/^#/s/^# \{0,1\}//p;}' "$SELF"; exit 0 ;; + *) echo "unknown arg: $1 (try --help)" >&2; exit 1 ;; + esac +done + +# Demo narration: what the step just established, in prose. Distinct from ok/ko, which +# are the machine-checked assertions. +note() { echo " -> $1"; } + +stop_port_forward() { + [ -n "$PF_PID" ] || return 0 + kill "$PF_PID" 2>/dev/null || true + wait "$PF_PID" 2>/dev/null || true + PF_PID="" +} + +# Only stops a kourier tunnel THIS script started -- ensure_port_forward echoes nothing when +# one was already up, and tearing down someone else's would break a concurrent session. +stop_kourier_port_forward() { + [ -n "$KOURIER_PF_PID" ] || return 0 + kill "$KOURIER_PF_PID" 2>/dev/null || true + wait "$KOURIER_PF_PID" 2>/dev/null || true + KOURIER_PF_PID="" +} + +# Is the harness itself answering? Distinguished from "the model is unreachable", which is +# what an empty /runs response would otherwise be blamed on -- they look identical at the +# verdict layer and have completely different fixes. Any HTTP response counts (even a 404): +# this is a transport check, not a health check. +wait_harness_reachable() { + local i + for ((i = 0; i < 20; i++)); do + # shellcheck disable=SC2086 # CURL_OPTS is intentionally word-split + if curl -s $CURL_OPTS -o /dev/null --max-time 5 ${CURL_HDR[@]+"${CURL_HDR[@]}"} "$BASE/" 2>/dev/null; then + return 0 + fi + sleep 2 + done + return 1 +} + +# A fresh credential per run, so the token in `docker inspect` is scoped to this one demo +# rather than being a constant checked into the repo. `head -c` leads the pipeline so no +# stage is killed by SIGPIPE (which pipefail would surface as a failure). +gen_relay_token() { + local t + t="$(openssl rand -hex 16 2>/dev/null || true)" + [ -n "$t" ] || t="$(head -c 16 /dev/urandom 2>/dev/null | od -An -tx1 | tr -d ' \n' || true)" + [ -n "$t" ] || abort "could not generate a random relay token (no openssl, and /dev/urandom is not readable). Set SANDBOX_TOKEN to supply one." + printf '%s' "$t" +} + +remove_worker_container() { + docker rm -f "$WORKER_CTR" >/dev/null 2>&1 || true +} + +remove_relay() { + kubectl delete -f relay-deployment.yaml --ignore-not-found --wait=true --timeout=60s >/dev/null 2>&1 || true +} + +# --teardown runs as its own process, so it cannot know whether some earlier invocation +# created $CLUSTER_NAME or whether it is the dev cluster --reuse-cluster was pointed at. +# Deleting it unconditionally would make a cleanup command destructive to something the demo +# never owned, so ask. --yes is the non-interactive answer; with no tty and no --yes, keep the +# cluster -- refusing to delete is the recoverable direction. +confirm_cluster_delete() { + [ "$ASSUME_YES" = 1 ] && return 0 + if [ ! -t 0 ]; then + echo " stdin is not a terminal, so cluster '$CLUSTER_NAME' is being KEPT." >&2 + echo " re-run with --yes to delete it, or: kind delete cluster --name $CLUSTER_NAME" >&2 + return 1 + fi + local reply="" + printf " delete kind cluster '%s'? Irreversible, and the demo may not have created it. [y/N] " "$CLUSTER_NAME" + read -r reply || true + case "$reply" in [yY] | [yY][eE][sS]) return 0 ;; *) return 1 ;; esac +} + +# Full teardown: everything this demo can create, in dependency order. Idempotent, so +# `--teardown` is safe to run against a half-finished or already-clean laptop. The cluster is +# the one exception to "remove everything" -- see confirm_cluster_delete. +full_teardown() { + echo "=== Teardown ===" + echo "--- stopping the tunnel ---"; stop_port_forward + echo "--- removing the worker container ---"; remove_worker_container + if kubectl cluster-info >/dev/null 2>&1; then + echo "--- restoring harness env ---"; restore_harness_env + echo "--- removing the relay ---"; remove_relay + fi + local cluster_removed=0 + if kind get clusters 2>/dev/null | grep -q "^${CLUSTER_NAME}$"; then + echo "--- kind cluster '$CLUSTER_NAME' ---" + if confirm_cluster_delete; then + kind delete cluster --name "$CLUSTER_NAME" >/dev/null 2>&1 || true + cluster_removed=1 + echo " deleted." + else + echo " kept." + fi + fi + echo "--- removing the built worker image ---" + docker rmi "$WORKER_IMAGE" >/dev/null 2>&1 || true + rm -f "$PF_LOG" 2>/dev/null || true + if [ "$cluster_removed" = 1 ]; then + echo "teardown complete: no cluster, container, or built image left behind" + else + echo "teardown complete: container, relay and built image removed; cluster '$CLUSTER_NAME' left running" + fi +} + +# EXIT trap. Restoring the harness env is the non-negotiable part: leaving it pointed at a +# selector matching nothing would break every later leaf run on this cluster. +cleanup() { + echo "" + if [ "$KEEP" = 1 ]; then + echo "--- Cleanup (--keep): restoring harness env, leaving relay + worker up ---" + restore_harness_env + stop_port_forward + stop_kourier_port_forward + echo "worker container '$WORKER_CTR' and the relay are still running." + echo "NOTE: the tunnel is down, so the worker's Attach stream is now closed. Restart it with:" + echo " kubectl port-forward -n $NS svc/sandbox-relay ${RELAY_PORT}:8443" + else + echo "--- Cleanup: restoring harness env, removing relay + worker ---" + restore_harness_env + stop_port_forward + remove_worker_container + remove_relay + stop_kourier_port_forward + if [ "$CREATED_CLUSTER" = 1 ]; then + echo "cluster '$CLUSTER_NAME' left running (this run created it). Remove everything with:" + echo " $0 --teardown" + else + # Never advertise --teardown as "remove everything" for a cluster that predates this + # run: a reader who reached teardown by following this hint has not re-read the README. + echo "cluster '$CLUSTER_NAME' predates this run and is untouched. Nothing else to clean up:" + echo "the relay, worker container and tunnel are already gone." + echo "('$0 --teardown' would offer to delete the cluster itself; it asks first.)" + fi + fi + rm -f "$PF_LOG" 2>/dev/null || true +} + +if [ "$TEARDOWN_ONLY" = 1 ]; then + full_teardown + exit 0 +fi + +# Echo the value of one env var on the harness ksvc (empty when unset). +ksvc_env_value() { + kubectl get ksvc "$KSVC" -n "$NS" -o json 2>/dev/null \ + | jq -r --arg n "$1" '.spec.template.spec.containers[0].env[]? | select(.name==$n) | .value // empty' 2>/dev/null || true +} + +# Start (or restart) the relay tunnel and wait for the local listener. +start_relay_port_forward() { + stop_port_forward + kubectl port-forward ${PF_ADDRESS_ARGS[@]+"${PF_ADDRESS_ARGS[@]}"} \ + -n "$NS" svc/sandbox-relay "${RELAY_PORT}:8443" >"$PF_LOG" 2>&1 & + PF_PID=$! + local i + for ((i = 0; i < 40; i++)); do + if (exec 3<>"/dev/tcp/127.0.0.1/${RELAY_PORT}") 2>/dev/null; then return 0; fi + sleep 0.25 + done + return 1 +} + +# Does a connection actually TRAVERSE the tunnel to a serving relay? A plain TCP connect to +# the forwarded port is not enough: kubectl accepts the local connection first and only then +# tries the pod, so a dead relay still yields a successful connect and a forwarding error +# logged afterwards. Checking the log is what distinguishes "relay is dead" from "the +# container cannot route here" -- two failures with completely different fixes. +relay_reachable_from_host() { + local before after + # Only errors logged by THIS probe count. Truncating the log would corrupt it (kubectl + # holds the fd and keeps writing at its old offset), so window it by line count instead. + before="$(wc -l < "$PF_LOG" 2>/dev/null || echo 0)" + (exec 3<>"/dev/tcp/127.0.0.1/${RELAY_PORT}") 2>/dev/null || return 1 + sleep 1 + after="$(tail -n "+$((before + 1))" "$PF_LOG" 2>/dev/null || true)" + ! grep -qE "error forwarding|connection refused|lost connection" <<<"$after" +} + +# Wait for the relay to actually serve. `rollout status` returns as soon as the pod is +# Running, but the entrypoint compiles the relay's TypeScript through tsx first -- measured +# ~4s between Running and the port being bound. Probing immediately is a race, and losing it +# looks exactly like a broken relay. A failed forward also kills kubectl port-forward +# outright ("lost connection to pod"), so restart the tunnel when it has died. +wait_relay_reachable_from_host() { + local attempts="${1:-30}" i + for ((i = 0; i < attempts; i++)); do + if [ -z "$PF_PID" ] || ! kill -0 "$PF_PID" 2>/dev/null; then + start_relay_port_forward || { sleep 2; continue; } + fi + if relay_reachable_from_host; then return 0; fi + sleep 2 + done + return 1 +} + +# Can a container reach the tunnel? Probed with the worker's own image and network config, +# via bash's /dev/tcp -- so this tests exactly what the worker will do, and needs no extra +# image pulled just to hold a netcat. +probe_from_container() { + docker run --rm --entrypoint bash "$@" "$WORKER_IMAGE" \ + -c "exec 3<>/dev/tcp/host.docker.internal/${RELAY_PORT}" >/dev/null 2>&1 +} + +echo "=== Remote-sandbox laptop demo (sandbox=$SANDBOX_ID, model=$MODEL) ===" +echo "Proving: a sandbox outside the cluster, with zero inbound rules, runs a leaf's tools." + +# --- 1. Preflight ---------------------------------------------------------------------- +claim "Preflight: tools present" +for tool in docker kind kubectl jq; do + command -v "$tool" >/dev/null 2>&1 || abort "$tool not found (needed for this demo)" +done +note "docker, kind, kubectl, jq found. No Go toolchain needed -- the worker's Dockerfile" +note "builds the binary in a golang builder stage." + +# --- 2. Cluster ------------------------------------------------------------------------ +claim "Cluster: Knative + Redis + harness + Alpine sandbox pool" +cluster_healthy() { + kubectl cluster-info >/dev/null 2>&1 \ + && kubectl get ksvc "$KSVC" -n "$NS" >/dev/null 2>&1 \ + && kubectl get deploy redis -n "$NS" >/dev/null 2>&1 +} +if [ "$REUSE_CLUSTER" = 1 ] && kind get clusters 2>/dev/null | grep -q "^${CLUSTER_NAME}$" \ + && kubectl config use-context "kind-${CLUSTER_NAME}" >/dev/null 2>&1 && cluster_healthy; then + note "reusing healthy cluster '$CLUSTER_NAME' (--reuse-cluster)" +else + # Check for the cluster BEFORE setup-kind.sh runs: it reuses an existing cluster rather than + # recreating it, so afterwards "the cluster exists" says nothing about who made it. + if kind get clusters 2>/dev/null | grep -q "^${CLUSTER_NAME}$"; then + note "cluster '$CLUSTER_NAME' already exists; setup-kind.sh will reuse it, and teardown will not offer to delete it as this run's own" + else + CREATED_CLUSTER=1 + fi + note "running setup-kind.sh (creates the cluster if absent; several minutes on a cold start)" + CLUSTER_NAME="$CLUSTER_NAME" ./setup-kind.sh \ + || abort "setup-kind.sh failed -- see its output above (a missing ANTHROPIC_API_KEY / ANTHROPIC_AUTH_TOKEN is the usual cause)" +fi +cluster_healthy || abort "cluster '$CLUSTER_NAME' is not serving the harness (ksvc/$KSVC or deploy/redis missing)" + +POOL_SELECTOR="$(resolve_pool_selector)" +# `|| true` so a failed query surfaces as the named abort below rather than as a bare +# set -e exit with no explanation of what went wrong. +POOL_POD_COUNT="$(count_pool_pods "$POOL_SELECTOR" || true)" +[ "$POOL_POD_COUNT" = "ERR" ] \ + && abort "could not query Running pods for pool selector '$POOL_SELECTOR' (kubectl failed -- wrong context, API error, or missing RBAC)" +[ "${POOL_POD_COUNT:-0}" -ge 1 ] \ + || abort "no Running sandbox pods match pool selector '$POOL_SELECTOR' -- the pod side of the A/B needs at least one" +SBOX_POD="$(first_pool_pod "$POOL_SELECTOR")" +note "harness ksvc ready; $POOL_POD_COUNT in-cluster sandbox pod(s) match '$POOL_SELECTOR' (sample: $SBOX_POD)" + +trap cleanup EXIT + +# The leaf requests below reach the harness over a kourier port-forward with a Host header +# (see lib.sh). Verify it answers now: every dispatch would otherwise return an empty body, +# which is indistinguishable from an unreachable model at the verdict layer. +claim "Harness endpoint: reachable" +KOURIER_PF_PID="$(ensure_port_forward || true)" +wait_harness_reachable \ + || abort "the harness is not answering on $BASE. A kourier port-forward is needed: kubectl port-forward -n kourier-system svc/kourier ${PORT}:80" +ok "harness answers on $BASE" + +# --- 3. Worker image ------------------------------------------------------------------- +claim "Worker image: built locally, for the HOST -- not loaded into kind" +docker build --load -f "$REPO_ROOT/remote-worker/Dockerfile" -t "$WORKER_IMAGE" "$REPO_ROOT" \ + >"${TMPDIR:-/tmp}/sh-demo-worker-build.log" 2>&1 \ + || abort "docker build of $WORKER_IMAGE failed (see ${TMPDIR:-/tmp}/sh-demo-worker-build.log)" +note "built $WORKER_IMAGE at host architecture." +note "No 'kind load' and no arch-matching with the kind node: this image never enters the" +note "cluster. It runs as a host container, which is the point of the whole demo." + +# --- 4. Relay -------------------------------------------------------------------------- +claim "Relay: the only thing the worker will dial" +kubectl apply -f relay-deployment.yaml >/dev/null || abort "kubectl apply relay-deployment.yaml failed" +if [ -n "$RELAY_TOKEN" ]; then + note "using the SANDBOX_TOKEN supplied in the environment." +else + RELAY_TOKEN="$(gen_relay_token)" + note "generated a random relay token for this run only (not relay-deployment.yaml's public 'dev-token')." +fi +# Patch the live Deployment BEFORE waiting on the rollout, so the pod that becomes Ready is +# already the one holding this run's token -- setting it afterwards would restart the relay +# out from under the readiness we just established. A later `kubectl apply -f +# relay-deployment.yaml` (relay-leaf-smoke.sh does exactly that) reverts it to the declared +# dev value, so this leaves nothing behind for other callers. +kubectl set env deploy/sandbox-relay -n "$NS" "SH_RELAY_TOKEN=$RELAY_TOKEN" >/dev/null \ + || abort "could not set SH_RELAY_TOKEN on deploy/sandbox-relay" +kubectl -n "$NS" rollout status deploy/sandbox-relay --timeout=90s >/dev/null \ + || abort "sandbox-relay rollout did not become ready" +note "relay up. It is inert until a worker attaches AND the harness is on SH_REMOTE_SANDBOX=1." + +# --- 5. Tunnel ------------------------------------------------------------------------- +claim "Tunnel: relay -> host, proven reachable from inside a container" +start_relay_port_forward || abort "kubectl port-forward svc/sandbox-relay did not start (see $PF_LOG)" +# Prove the relay is genuinely serving BEFORE testing container networking, so a dead relay +# is never misreported as a networking problem. +wait_relay_reachable_from_host 30 \ + || diagnose_relay_crash "the tunnel is up but nothing answered on the relay's :8443 within 60s" +ok "relay answers through the tunnel from the host" +if probe_from_container; then + note "host.docker.internal reaches the default 127.0.0.1 port-forward bind (Docker Desktop" + note "or a podman machine). Nothing is exposed beyond the host." +elif probe_from_container --add-host=host.docker.internal:host-gateway; then + WORKER_DOCKER_ARGS+=(--add-host=host.docker.internal:host-gateway) + note "reachable via an explicit --add-host=host.docker.internal:host-gateway mapping." +else + # Native Linux Docker: host-gateway is the bridge IP, which a loopback-only + # port-forward does not answer on. Widen the bind, and say so -- for the demo's + # duration the relay port is reachable from the local network. + PF_ADDRESS_ARGS=(--address 0.0.0.0) + note "loopback bind unreachable from a container; retrying with --address 0.0.0.0" + start_relay_port_forward || abort "kubectl port-forward --address 0.0.0.0 did not start (see $PF_LOG)" + if probe_from_container --add-host=host.docker.internal:host-gateway; then + WORKER_DOCKER_ARGS+=(--add-host=host.docker.internal:host-gateway) + echo " WARN: the relay port is bound to 0.0.0.0 for the duration of this demo, so it is" + echo " reachable from your local network. It accepts this run's randomly generated" + echo " bearer token only -- not relay-deployment.yaml's public 'dev-token' -- so a" + echo " LAN peer cannot Attach as a sandbox with a credential read from the repo." + echo " It is torn down on exit." + else + abort "the relay answers from the host, but no container could reach it on port $RELAY_PORT -- this is container-to-host networking, not the relay. Alternative: attach the worker to kind's docker network and dial the node directly (see README-worker.md)." + fi +fi + +# --- 6. Worker ------------------------------------------------------------------------- +claim "Worker: a host container with NO published ports" +remove_worker_container +WORKER_RUN=(docker run -d --name "$WORKER_CTR" + ${WORKER_DOCKER_ARGS[@]+"${WORKER_DOCKER_ARGS[@]}"} + -e "SANDBOX_ID=$SANDBOX_ID" + -e "RELAY_ADDR=host.docker.internal:${RELAY_PORT}" + -e "SANDBOX_TOKEN=$RELAY_TOKEN" + "$WORKER_IMAGE") +echo " \$ ${WORKER_RUN[*]}" +"${WORKER_RUN[@]}" >/dev/null || abort "docker run of the worker container failed" +note "Note what is absent: no -p, no --publish, no inbound rule. The worker dials OUT." +note "Its whole credential set is a bearer token, scoped to this run -- no LLM key, no kubeconfig." + +# --- 7. Presence ----------------------------------------------------------------------- +claim "Presence: the worker's live Attach stream IS its registration" +assert_presence "$SANDBOX_ID" || abort "worker never registered -- check 'docker logs $WORKER_CTR' (a token mismatch with the relay's SH_RELAY_TOKEN is rejected fail-closed)" +note "Redis holds a record only while the stream is open; nothing polls, nothing heartbeats a URL." + +# --- 8. Discriminator ------------------------------------------------------------------ +claim "Discriminator: verify Alpine vs RHEL BEFORE relying on it" +POD_OS="$(kubectl exec "$SBOX_POD" -n "$NS" -- cat /etc/os-release 2>/dev/null || true)" +WORKER_OS="$(docker exec "$WORKER_CTR" cat /etc/os-release 2>/dev/null || true)" +validate_discriminator "$POD_OS" "$WORKER_OS" "$SBOX_POD" "$WORKER_CTR (host container)" +note "Now a verdict on the pattern 'Alpine' identifies which filesystem answered." + +# --- 9. Run A: the pod path ------------------------------------------------------------ +claim "Run A -- in-cluster Alpine sandbox pod" +[ "$(ksvc_env_value SH_REMOTE_SANDBOX)" = "1" ] \ + && abort "the harness is already on SH_REMOTE_SANDBOX=1, so run A would not be the pod path. Run '$0 --teardown' first, or restore the ksvc env by hand." +snapshot_harness_env +RESP_A="$(dispatch_pattern "demo-pod-$$" "Alpine")" +assert_verdict "A/pod-path/Alpine" "$RESP_A" "FLAGGED" \ + "the in-cluster pool is Alpine, so a leaf grepping for 'Alpine' must be FLAGGED here; an empty verdict usually means the model endpoint is unreachable" +REASON_A="$(verdict_reason "$RESP_A")" +[ -n "$REASON_A" ] && note "model's reason: $REASON_A" + +# --- 10. Flip -------------------------------------------------------------------------- +claim "Flip to the remote path (and make a pod win IMPOSSIBLE)" +assert_no_pods_match "$REMOTE_ONLY_SELECTOR" +note "This is the trap the issue warns about: SH_REMOTE_SANDBOX=1 alone would leave idle pods" +note "in the candidate set, and a pod could win the lease -- proving nothing while looking green." +flip_harness_env SH_REMOTE_SANDBOX=1 SH_RELAY_ADDR="$IN_CLUSTER_RELAY_ADDR" \ + KAGENTI_SANDBOX_POOL_SELECTOR="$REMOTE_ONLY_SELECTOR" +wait_latest_ready 150 || abort "harness did not reach a ready latest revision after the flip" +note "harness -> relay via $IN_CLUSTER_RELAY_ADDR (in-cluster DNS);" +note "worker -> relay via host.docker.internal:${RELAY_PORT} (outbound through the tunnel)." + +# --- 11. Run B: the remote path -------------------------------------------------------- +claim "Run B -- remote host container (both directions asserted)" +RESP_B="$(dispatch_pattern "demo-remote-alpine-$$" "Alpine")" +assert_verdict "B/remote/Alpine" "$RESP_B" "CLEAR" \ + "FLAGGED here means the exec landed on an Alpine sandbox pod, not the remote RHEL worker -- the demo would be proving nothing" +REASON_B="$(verdict_reason "$RESP_B")" +[ -n "$REASON_B" ] && note "model's reason: $REASON_B" +RESP_B2="$(dispatch_pattern "demo-remote-redhat-$$" "Red Hat")" +assert_verdict "B/remote/RedHat" "$RESP_B2" "FLAGGED" \ + "CLEAR here means the exec landed on an Alpine sandbox pod, not the remote RHEL worker -- the demo would be proving nothing" +note "Asserted both ways: a pod-landed exec fails one check or the other, never neither." + +# --- 12. Summary ----------------------------------------------------------------------- +echo "" +echo "=== The A/B ===" +printf '%-34s | %-8s | %s\n' "backend" "'Alpine'" "model's stated reason" +printf '%-34s-+-%-8s-+-%s\n' "----------------------------------" "--------" "---------------------" +printf '%-34s | %-8s | %s\n' "in-cluster pod ($SBOX_POD)" \ + "$(jq -r '.verdict.verdict // "?"' <<<"$RESP_A" 2>/dev/null || echo '?')" "${REASON_A:-(none returned)}" +printf '%-34s | %-8s | %s\n' "remote host container" \ + "$(jq -r '.verdict.verdict // "?"' <<<"$RESP_B" 2>/dev/null || echo '?')" "${REASON_B:-(none returned)}" +echo "" +echo "Same request, opposite verdicts. The second one ran on a container on this host that" +echo "the cluster cannot reach and never authenticated to -- it dialed out, and nothing else." + +echo ""; echo "=== Results: $PASS passed, $FAIL failed ===" +if [ "$FAIL" -gt 0 ]; then echo "DEMO FAIL"; exit 1; else echo "DEMO PASS"; exit 0; fi diff --git a/deploy/knative/lib-relay.sh b/deploy/knative/lib-relay.sh new file mode 100644 index 0000000..5fa2441 --- /dev/null +++ b/deploy/knative/lib-relay.sh @@ -0,0 +1,260 @@ +#!/usr/bin/env bash +# deploy/knative/lib-relay.sh +# Shared assertions for the remote-sandbox (gRPC relay + worker) proofs. +# +# Sourced by BOTH the gated conformance smoke (relay-leaf-smoke.sh, worker as an +# in-cluster pod) and the laptop demo (demo-remote-worker.sh, worker as a host +# container). Everything here is worker-topology-agnostic on purpose: the two callers +# differ only in WHERE the worker runs and HOW its /etc/os-release is read, so the +# assertions that decide whether the proof holds must not be duplicated between them. +# A drifting copy would let one path keep asserting something the other no longer does. +# +# Source AFTER lib.sh -- this file builds on NS/KSVC/BASE/CURL_OPTS/CURL_HDR and ok/ko. +# +# shellcheck shell=bash + +# Model used for the leaf's verdict call. Owned here so both callers agree. +MODEL="${MODEL:-${SH_MODEL:-claude-haiku-4-5}}" + +# Harness env flip bookkeeping. Callers must not set these directly: snapshot_harness_env +# fills SH_ENV_SNAPSHOT, flip_harness_env raises SH_ENV_FLIPPED, restore_harness_env +# clears it. Kept as globals (not passed around) so an EXIT trap can restore with no args. +SH_ENV_SNAPSHOT="" +SH_ENV_FLIPPED=0 + +# --- Output helpers ------------------------------------------------------------------- +# claim: announce the step about to be proven. abort: unrecoverable, exit non-zero. +claim() { echo ""; echo "--- $1 ---"; } +abort() { echo "ABORT: $1" >&2; exit 1; } + +# --- Revision readiness ---------------------------------------------------------------- +# Wait until the ksvc's latest-created revision is also its latest-ready revision (or +# timeout). lib.sh's wait_ksvc_ready swallows failures by design (`|| true`); this adds a +# hard check specifically for the flip/restore transitions, where serving the wrong +# revision would mean asserting against the wrong backend. +# Usage: wait_latest_ready [timeoutSec] +wait_latest_ready() { + local timeout="${1:-150}" waited=0 created ready + while [ "$waited" -lt "$timeout" ]; do + created="$(kubectl get ksvc "$KSVC" -n "$NS" -o jsonpath='{.status.latestCreatedRevisionName}' 2>/dev/null || true)" + ready="$(kubectl get ksvc "$KSVC" -n "$NS" -o jsonpath='{.status.latestReadyRevisionName}' 2>/dev/null || true)" + if [ -n "$created" ] && [ "$created" = "$ready" ]; then + echo " ksvc/$KSVC latest-ready revision: $ready" + return 0 + fi + sleep 3; waited=$((waited + 3)) + done + return 1 +} + +# --- Pool selector introspection ------------------------------------------------------- +# Echo the pool selector the harness is currently configured with, falling back to the +# setup-kind.sh/setup-ocp.sh default when the env var is absent. +resolve_pool_selector() { + local sel + sel="$(kubectl get ksvc "$KSVC" -n "$NS" -o json 2>/dev/null \ + | jq -r '.spec.template.spec.containers[0].env[]? | select(.name=="KAGENTI_SANDBOX_POOL_SELECTOR") | .value' 2>/dev/null || true)" + echo "${sel:-sh.kagenti.io/sandbox-pool=default}" +} + +# Echo the number of Running pods matching a label selector, or "ERR" (return 1) when the +# query itself failed. The distinction matters: piping a failed `kubectl get` into `wc -l` +# yields 0, which is indistinguishable from "the selector genuinely matches nothing" -- and +# a wrong context, transient API error, expired credential or missing RBAC would then read +# as an empty candidate set. That makes assert_no_pods_match below fail OPEN, which is the +# one direction it must not. Callers must treat "ERR" as fatal, never as zero. +# `grep -c .` rather than `wc -l` so a trailing newline is not counted as a pod. +# Usage: count_pool_pods +count_pool_pods() { + local out + out="$(kubectl get pods -n "$NS" -l "$1" --field-selector=status.phase=Running --no-headers 2>/dev/null)" \ + || { echo "ERR"; return 1; } + printf '%s' "$out" | grep -c . | tr -d ' ' +} + +# Echo the name of the first Running pod matching a label selector (empty if none). +# Usage: first_pool_pod +first_pool_pod() { + kubectl get pods -n "$NS" -l "$1" --field-selector=status.phase=Running --no-headers 2>/dev/null \ + | awk 'NR==1{print $1}' +} + +# Assert a selector matches ZERO Running pods. This is what makes the "exec landed on a +# pod" trap structurally impossible rather than merely detectable: select-sandbox.ts +# builds candidates = [...pods, ...grpcRecs], so with no pods in the candidate set its +# least-loaded-first leasing has nothing to route around the worker with. Abort (not ko) +# on a miss -- the remote assertions would silently prove nothing. +# Usage: assert_no_pods_match +assert_no_pods_match() { + local sel="$1" n + n="$(count_pool_pods "$sel" || true)" + # A failed query must abort, not pass. Claiming "matches 0 Running pods" on the strength of + # a kubectl error would hand back exactly the vacuous green this assertion exists to rule + # out: the remote assertions would then run against an unverified candidate set. + [ "$n" = "ERR" ] && abort "could not determine whether pool selector '$sel' matches any Running pods -- the kubectl query failed (wrong context, API error, or missing RBAC). Refusing to continue: an unverified candidate set makes the remote proof vacuous." + if [ "${n:-0}" -eq 0 ]; then + ok "pool selector '$sel' matches 0 Running pods -- the remote worker is the only lease candidate" + else + abort "pool selector '$sel' matches $n Running pod(s); a pod could win the lease and the remote proof would be vacuous. Refusing to continue." + fi +} + +# --- Presence (the worker's live Attach stream IS its registration) -------------------- +# Poll Redis until the sandbox id appears with transport=grpc. Registration happens +# asynchronously after the worker starts -- a worker has no readiness signal an HTTP/TCP +# probe could observe, so "the process is up" never means "it has registered". +# Usage: assert_presence [attempts] +assert_presence() { + local sid="$1" attempts="${2:-20}" presence="" i + for ((i = 0; i < attempts; i++)); do + presence="$(kubectl exec deploy/redis -n "$NS" -- redis-cli HGETALL sh:sandbox:records 2>/dev/null || true)" + if echo "$presence" | grep -qF "$sid" && echo "$presence" | grep -q '"transport":"grpc"'; then + ok "worker $sid present in sh:sandbox:records with transport=grpc" + return 0 + fi + sleep 2 + done + ko "worker $sid not found (or wrong transport) in sh:sandbox:records; presence dump: $(echo "$presence" | head -c 300)" + return 1 +} + +# Poll Redis until the sandbox id is GONE -- the stream-close teardown path. +# Usage: assert_presence_gone [attempts] +assert_presence_gone() { + local sid="$1" attempts="${2:-20}" i + for ((i = 0; i < attempts; i++)); do + if ! kubectl exec deploy/redis -n "$NS" -- redis-cli HGETALL sh:sandbox:records 2>/dev/null | grep -qF "$sid"; then + ok "presence record for $sid cleared when the worker's Attach stream closed" + return 0 + fi + sleep 2 + done + ko "presence record for $sid survived the worker going away (stream-close teardown did not propagate)" + return 1 +} + +# --- Relay health ----------------------------------------------------------------------- +# Diagnose a relay that is Running but not serving, and abort with the cause. `kubectl +# rollout status` returns Ready as soon as the pod is Running -- relay-deployment.yaml +# declares no readinessProbe, because a relay's real readiness is "a worker's Attach stream +# is parked here", which no HTTP/TCP probe could observe. So a relay that dies before +# binding :8443 still passes rollout, and the failure resurfaces much later as an +# inexplicable connection error pointing at the network instead of at the relay. +# +# Call this the moment something cannot reach the relay, so the cause is named where it is +# still legible. Usage: diagnose_relay_crash +diagnose_relay_crash() { + local ctx="$1" pod restarts last + pod="$(kubectl get pods -n "$NS" -l app=sandbox-relay -o jsonpath='{.items[0].metadata.name}' 2>/dev/null || true)" + [ -n "$pod" ] || abort "$ctx -- and there is no sandbox-relay pod at all (was relay-deployment.yaml applied?)" + restarts="$(kubectl get pod "$pod" -n "$NS" -o jsonpath='{.status.containerStatuses[0].restartCount}' 2>/dev/null || echo 0)" + last="$(kubectl get pod "$pod" -n "$NS" -o jsonpath='{.status.containerStatuses[0].lastState.terminated.reason}' 2>/dev/null || true)" + # OOMKilled is the failure this repo has actually hit: a 128Mi limit against ~225 MiB of + # node+tsx startup. Name it and its fix rather than reporting a generic crash. + if [ "$last" = "OOMKilled" ]; then + abort "$ctx -- relay pod $pod is being OOMKilled (restarts=$restarts), so it never bound :8443. Raise the memory limit in relay-deployment.yaml: node+tsx needs ~225 MiB just to idle." + fi + if [ "${restarts:-0}" -gt 0 ]; then + abort "$ctx -- relay pod $pod has restarted $restarts time(s) (lastState=${last:-unknown}); it is not serving. Inspect: kubectl logs $pod -n $NS --previous" + fi + abort "$ctx -- relay pod $pod reports no restarts, so the relay process is up but unreachable on the path tried. Inspect: kubectl logs $pod -n $NS" +} + +# --- Discriminator --------------------------------------------------------------------- +# Verify the Alpine/RHEL fingerprint BEFORE anything relies on it. The in-cluster sandbox +# pool runs Alpine (sandbox-pool.yaml); the worker image runs RHEL +# (registry.access.redhat.com/ubi9/ubi-minimal). A leaf grepping /etc/os-release for +# "Alpine" is therefore FLAGGED on a pod and CLEAR on the worker, and the reverse for +# "Red Hat" -- so asserting BOTH catches a pod-landed exec either way. +# +# Takes the two os-release texts as STRINGS rather than fetching them: the gate reads the +# worker via `kubectl exec` and the demo via `docker exec`, but the assertion that decides +# whether the discriminator is trustworthy must be identical. +# Usage: validate_discriminator +validate_discriminator() { + local pod_os="$1" worker_os="$2" pod_label="$3" worker_label="$4" + if echo "$pod_os" | grep -qi 'Alpine' && ! echo "$pod_os" | grep -qi 'Red Hat' \ + && echo "$worker_os" | grep -qi 'Red Hat' && ! echo "$worker_os" | grep -qi 'Alpine'; then + ok "discriminator holds: sandbox pod ($pod_label)=Alpine, worker ($worker_label)=Red Hat" + else + abort "discriminator invalid -- sandbox pod /etc/os-release: [$pod_os]; worker /etc/os-release: [$worker_os]. Refusing to run assertions that would be meaningless without a verified discriminator." + fi +} + +# --- Leaf dispatch + verdict assertion ------------------------------------------------- +# dispatch_pattern -> echoes terminal JSON from POST /runs, grepping +# /etc/os-release for . Mirrors leaf-smoke.sh's dispatch_item curl invocation. +dispatch_pattern() { + local sid="$1" pat="$2" body + body=$(jq -nc --arg s "$sid" --arg m "$MODEL" --arg p "$pat" \ + '{sessionId:$s, model:$m, item:{item_id:"i1", file:"/etc/os-release", pattern:$p}}') + # shellcheck disable=SC2086 # CURL_OPTS is intentionally word-split + # `|| true`: a connection-level failure (timeout, connection refused) must not exit + # the caller under set -e here -- it should instead yield an empty body so + # assert_verdict's "model endpoint unreachable" hint is reached instead of bypassed. + curl -s $CURL_OPTS --max-time 120 ${CURL_HDR[@]+"${CURL_HDR[@]}"} \ + -H "Content-Type: application/json" -d "$body" "$BASE/runs" || true +} + +# assert_verdict