Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 11 additions & 1 deletion Makefile
Original file line number Diff line number Diff line change
@@ -1,4 +1,5 @@
.PHONY: lint fmt test test-deploy typecheck demo-remote-sandbox demo-remote-sandbox-teardown
.PHONY: lint fmt test test-deploy typecheck demo-remote-sandbox demo-remote-sandbox-teardown \
demo-promoted-workflow demo-promoted-workflow-teardown

lint:
pre-commit run --all-files
Expand Down Expand Up @@ -31,3 +32,12 @@ demo-remote-sandbox:

demo-remote-sandbox-teardown:
bash deploy/knative/demo-remote-worker.sh --teardown

# Promote a Claude Code workflow authored in a minimal local sandbox, then prove it ran remotely.
# Needs a warm cluster whose image contains the promotion feature; the script gates on that.
# See docs/demos/promoted-workflow-demo.md.
demo-promoted-workflow:
bash deploy/knative/demo-promoted-workflow.sh $(DEMO_ARGS)

demo-promoted-workflow-teardown:
bash deploy/knative/demo-promoted-workflow.sh --teardown
366 changes: 366 additions & 0 deletions deploy/knative/demo-promoted-workflow.sh

Large diffs are not rendered by default.

8 changes: 8 additions & 0 deletions deploy/knative/fixtures/promoted-demo/CLAUDE.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
# Release engineering house rules

These rules apply to every ship note, without exception.

- Before writing a ship note you MUST load the `ship-note` skill with the read tool, and follow its
format exactly — including the release token it names.
- Every ship note ends with a `RISK:` line, rated `low`, `medium` or `high`, with a reason.
- **Never invent an incident id.** Cite the id recorded in memory, or write `no linked incident`.
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
Write the ship note for the fix described below, following the house rules exactly.
1 change: 1 addition & 0 deletions deploy/knative/fixtures/promoted-demo/memory/MEMORY.md
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
- [Auth timeout incident](auth-timeout-incident.md) — the idle reaper that closed live sessions, and the ticket tracking it
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
---
name: auth-timeout-incident
description: The auth timeout regression, its root cause, and the ticket that tracks it
metadata:
type: project
---

Sessions dropped after 30 seconds of inactivity because the idle reaper measured time since the
connection _opened_ rather than time since the last byte. Tracked as **KAG-4471**.

Fixed by resetting the reaper clock on every frame. The fix ships behind the
`auth.idle_reaper_v2` flag, so it is reversible without a redeploy.
23 changes: 23 additions & 0 deletions deploy/knative/fixtures/promoted-demo/skills/ship-note/SKILL.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,23 @@
---
name: ship-note
description: Use when writing a ship note, release note or changelog entry for a fix that is shipping — emits the house SHIP NOTE block.
---

# Ship note

Reply with exactly this block, and nothing else:

```
SHIP NOTE · <slug>
WHAT: <one line, past tense>
WHY: <the user-visible symptom>
TICKET: <the incident id from memory, or "no linked incident">
RISK: <low|medium|high> — <reason>
TOKEN: <the token from references/release-token.md>
```

The token is **not** in this file. Read `references/release-token.md` from this skill's own
directory and copy the token verbatim into the `TOKEN:` line.

Do not guess the token. If you cannot read that file, write `TOKEN: unavailable` — a wrong token is
worse than an absent one.
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
SHIPNOTE-7F3A-SANDBOX-OK
136 changes: 136 additions & 0 deletions deploy/knative/tests/demo-promoted-workflow.test.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,136 @@
#!/usr/bin/env bash
# deploy/knative/tests/demo-promoted-workflow.test.sh
#
# Cluster-free tests for the promoted-workflow demo. The demo itself needs a cluster and two real
# model calls, so it cannot run on every PR -- but its FIXTURE can rot silently, and a rotted
# fixture turns the demo into a green run that proves nothing. Specifically:
#
# - the A/B hinges on one unguessable string (the ticket id) living ONLY in the memory file, and
# on one token living ONLY in the skill's references/ subdirectory. If an edit moves either into
# CLAUDE.md or SKILL.md, the bare arm could produce it from the prompt alone and the demo's
# central claim quietly becomes untestable.
# - the constants the script greps for must match the fixture's contents. A typo in either makes
# every claim fail against a working cluster, which reads as a broken feature.
#
# No cluster required. Run: bash deploy/knative/tests/demo-promoted-workflow.test.sh
set -uo pipefail

DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
FIX="$DIR/fixtures/promoted-demo"
SCRIPT="$DIR/demo-promoted-workflow.sh"
fails=0
check() { if [ "$2" = "$3" ]; then echo " ok: $1"; else
echo " FAIL: $1 (want '$3', got '$2')"
fails=$((fails + 1))
fi; }

echo "== fixture layout"
for f in CLAUDE.md skills/ship-note/SKILL.md skills/ship-note/references/release-token.md \
commands/ship-note.md memory/MEMORY.md memory/auth-timeout-incident.md; do
check "$f exists" "$([ -f "$FIX/$f" ] && echo yes || echo no)" "yes"
done

echo "== the script's constants match the fixture"
TICKET=$(grep -E '^TICKET=' "$SCRIPT" | head -1 | cut -d'"' -f2)
TOKEN=$(grep -E '^TOKEN=' "$SCRIPT" | head -1 | cut -d'"' -f2)
check "TICKET is set" "$([ -n "$TICKET" ] && echo yes || echo no)" "yes"
check "TOKEN is set" "$([ -n "$TOKEN" ] && echo yes || echo no)" "yes"
check "the ticket is in the memory file" \
"$(grep -q "$TICKET" "$FIX/memory/auth-timeout-incident.md" && echo yes || echo no)" "yes"
check "the token is the release-token file's content" \
"$(tr -d '[:space:]' < "$FIX/skills/ship-note/references/release-token.md")" "$TOKEN"

echo "== the unguessable strings leak nowhere else in the fixture"
# The ticket must not be reachable from anything the harness injects into the system prompt, or the
# bare arm gets it for free and stops being a control.
for f in CLAUDE.md skills/ship-note/SKILL.md commands/ship-note.md memory/MEMORY.md; do
check "$f does not contain the ticket" \
"$(grep -q "$TICKET" "$FIX/$f" && echo leaked || echo clean)" "clean"
done
# The token must exist ONLY in references/, so producing it requires a sandbox read.
check "SKILL.md does not contain the token" \
"$(grep -q "$TOKEN" "$FIX/skills/ship-note/SKILL.md" && echo leaked || echo clean)" "clean"
check "CLAUDE.md does not contain the token" \
"$(grep -q "$TOKEN" "$FIX/CLAUDE.md" && echo leaked || echo clean)" "clean"
check "the prompt the script sends does not contain the ticket" \
"$(grep -E '^PROMPT=' "$SCRIPT" | grep -q "$TICKET" && echo leaked || echo clean)" "clean"

echo "== the skill is loadable by pi's resolver"
# resolve.ts reads the bare frontmatter `name`; classify.ts dedupes on it. A missing or namespaced
# name silently changes the bundle path the script asserts against.
check "SKILL.md has frontmatter name: ship-note" \
"$(awk '/^name:/{print $2; exit}' "$FIX/skills/ship-note/SKILL.md")" "ship-note"
check "SKILL.md has a description (required by the Agent Skills spec)" \
"$(grep -cE '^description:' "$FIX/skills/ship-note/SKILL.md")" "1"
# pi puts only name+description in the system prompt and tells the model to read the body, so the
# description is what decides whether the skill is ever loaded at all.
check "the description mentions a ship note, so the task matches it" \
"$(grep -iE '^description:.*ship note' "$FIX/skills/ship-note/SKILL.md" >/dev/null && echo yes || echo no)" "yes"

echo "== SKILL.md instructs the sibling read the demo asserts"
check "SKILL.md names references/release-token.md" \
"$(grep -q 'references/release-token.md' "$FIX/skills/ship-note/SKILL.md" && echo yes || echo no)" "yes"
check "SKILL.md tells the model to fail honestly rather than guess" \
"$(grep -q 'unavailable' "$FIX/skills/ship-note/SKILL.md" && echo yes || echo no)" "yes"

echo "== MEMORY.md links resolve (a dangling link is a preflight warning)"
while read -r target; do
[ -z "$target" ] && continue
check "MEMORY.md link '$target' exists" \
"$([ -f "$FIX/memory/$target" ] && echo yes || echo no)" "yes"
done < <(grep -oE '\]\([^)]+\.md\)' "$FIX/memory/MEMORY.md" | sed 's/](//; s/)//')

echo "== the script's own guards"
check "refuses a sandbox dir it did not create (marker gate)" \
"$(grep -q 'REFUSING to touch' "$SCRIPT" && echo yes || echo no)" "yes"
check "does not default the redis forward to 6379" \
"$(grep -qE '^REDIS_PORT=.*:-6379\}' "$SCRIPT" && echo bad || echo ok)" "ok"
check "gates on the deployed image implementing configRef" \
"$(grep -q 'unknown digest' "$SCRIPT" && echo yes || echo no)" "yes"
check "purges the shared sandbox cache before the control run" \
"$(grep -q 'sh-config' "$SCRIPT" && echo yes || echo no)" "yes"
check "verifies the upload landed in the cluster's redis" \
"$(grep -q 'config:bundle:' "$SCRIPT" && echo yes || echo no)" "yes"

echo "== --teardown is sandbox-scoped and touches no cluster"
# Same property demo-teardown-scope.test.sh pins for the remote-worker demo: a teardown must not
# delete infrastructure it did not create. This demo's teardown removes a /tmp sandbox and nothing
# else, so any kind/kubectl call on that path is a regression. kind/kubectl/docker are mocked and
# only the call log is asserted.
TMP="$(mktemp -d)"
trap 'rm -rf "$TMP"' EXIT
export MOCK_LOG="$TMP/calls.log"
: > "$MOCK_LOG"
mkdir -p "$TMP/bin"
for b in kind kubectl docker pnpm; do
cat > "$TMP/bin/$b" <<EOF
#!/usr/bin/env bash
echo "$b \$*" >> "\$MOCK_LOG"
exit 0
EOF
chmod +x "$TMP/bin/$b"
done

# A sandbox carrying the marker: teardown must remove exactly this, via plain rm, not via a cluster.
SBX="$TMP/sandbox"
mkdir -p "$SBX"
touch "$SBX/.sh-demo-sandbox"
PATH="$TMP/bin:$PATH" SH_DEMO_SANDBOX="$SBX" bash "$SCRIPT" --teardown > "$TMP/out" 2>&1
check "teardown exits 0" "$?" "0"
check "the marked sandbox is gone" "$([ -e "$SBX" ] && echo present || echo gone)" "gone"
check "no kind calls" "$(grep -c '^kind ' "$MOCK_LOG")" "0"
check "no kubectl calls" "$(grep -c '^kubectl ' "$MOCK_LOG")" "0"

# An unmarked directory must survive: this is the guard against a mistyped SH_DEMO_SANDBOX.
UNMARKED="$TMP/not-ours"
mkdir -p "$UNMARKED"
touch "$UNMARKED/precious.txt"
PATH="$TMP/bin:$PATH" SH_DEMO_SANDBOX="$UNMARKED" bash "$SCRIPT" --teardown > "$TMP/out2" 2>&1
check "an unmarked directory is left alone" \
"$([ -f "$UNMARKED/precious.txt" ] && echo kept || echo DELETED)" "kept"

if [ "$fails" -ne 0 ]; then
echo "FAILED: $fails check(s)"
exit 1
fi
echo "PASS"
9 changes: 5 additions & 4 deletions docs/demos/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,10 +6,11 @@ performed live in front of an audience.

## What lives here

| Demo | Shows | Time |
| ------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- |
| [`serverless-harness-demo.md`](./serverless-harness-demo.md) | An agent that **scales to a true zero**, resumes from cold with full memory, then **fans out into a worker fleet** that appears on demand and vanishes when the queue drains | ~10 min |
| [`remote-sandbox-demo.md`](./remote-sandbox-demo.md) | A **sandbox outside the cluster** with zero inbound rules, executing a leaf's tool calls — one free-form prompt that names a different OS on each backend, and a secret planted by hand that the cluster reads back | ~10 min |
| Demo | Shows | Time |
| ------------------------------------------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- |
| [`serverless-harness-demo.md`](./serverless-harness-demo.md) | An agent that **scales to a true zero**, resumes from cold with full memory, then **fans out into a worker fleet** that appears on demand and vanishes when the queue drains | ~10 min |
| [`remote-sandbox-demo.md`](./remote-sandbox-demo.md) | A **sandbox outside the cluster** with zero inbound rules, executing a leaf's tool calls — one free-form prompt that names a different OS on each backend, and a secret planted by hand that the cluster reads back | ~10 min |
| [`promoted-workflow-demo.md`](./promoted-workflow-demo.md) | A Claude Code workflow authored on your laptop — one skill, a `CLAUDE.md`, one memory file — **running unchanged in the cluster**: the same prompt twice, one field apart, and only the promoted run can cite an incident id that exists nowhere but your memory directory | ~8 min |

## Demo vs. smoke test vs. spec

Expand Down
Loading
Loading