Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -15,8 +15,9 @@ test:
# Cluster-free unit tests for the deploy/ shell scripts: kubectl, kind and docker are
# mocked on PATH and only the call log is asserted. Run in CI by the `deploy-scripts` job.
# `set -e` so one failing test file fails the target instead of being scrolled past.
# deploy/claude/tests covers the /promote slash-command asset, which nothing else type-checks.
test-deploy:
@set -e; for t in deploy/knative/tests/*.test.sh; do echo "== $$t"; bash "$$t"; done
@set -e; for t in deploy/knative/tests/*.test.sh deploy/claude/tests/*.test.sh; do echo "== $$t"; bash "$$t"; done

typecheck:
cd harness && pnpm exec tsc --noEmit
Expand Down
18 changes: 18 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -186,6 +186,24 @@ troubleshooting — is in **[`deploy/knative/README-ocp.md`](deploy/knative/READ
Iterate on a workflow locally in Claude Code — skills, `CLAUDE.md`, memory, a slash command —
then promote it:

From inside Claude Code, in the project you want to promote:

```bash
mkdir -p ~/.claude/commands # once
cp deploy/claude/commands/promote.md ~/.claude/commands/ # once
export SH_HARNESS_DIR=/path/to/serverless-harness # once, per shell

/promote my-workflow # in any project, from Claude Code
```

`/promote` sets the `HOME` override and `--project` for you, checks that the repo boundary and the
Redis tunnel are right before uploading, and reads the digest back through the cluster's own client
afterwards. Install it into your **real** `~/.claude/commands/`: with `HOME` pointed at the project,
`promote` bundles every prompt in that project's `.claude/commands/`, so a `/promote` living there
would ship itself into every bundle.

Or drive the CLI directly:

```bash
cd harness && pnpm promote --entry my-workflow --project /path/to/your/project
```
Expand Down
98 changes: 98 additions & 0 deletions deploy/claude/commands/promote.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,98 @@
---
description: Promote this project's Claude Code workflow into the serverless harness
argument-hint: <entry-prompt> [--dry-run] [--deny <skill>] [--sandbox-image <ref>]
allowed-tools: Bash(pnpm:*), Bash(kubectl port-forward:*), Bash(kubectl exec:*), Bash(test:*), Bash(ls:*), Bash(lsof:*), Bash(pwd), Bash(echo:*), Bash(git init:*)
---

<!--
allowed-tools notes, so a later edit does not widen this back:
* kubectl is granted per verb (port-forward, exec) rather than as `kubectl:*`, because this file is
installed into real user scope and is therefore visible in every project -- `kubectl:*` would
carry `kubectl delete` with it.
* pnpm stays broad on purpose. The documented invocation begins with env assignments
(`HOME="$PWD" REDIS_URL=... pnpm --dir ...`), so a narrower `Bash(pnpm --dir:*)` prefix is not
reliably matched, and the env has to be inline because exports do not survive between tool calls.
* echo is granted because five of the six Context probes end in `&& echo`/`|| echo`.
* jq and `git rev-parse` were granted but never invoked, and are gone. `git init` stays: guard 1
offers it.
-->

## Context

- Project being promoted (this is what travels): !`pwd`
- Harness checkout (`$SH_HARNESS_DIR`): !`echo "${SH_HARNESS_DIR:-UNSET}"`
- Harness CLI present: !`test -f "${SH_HARNESS_DIR:-/nonexistent}/harness/package.json" && echo yes || echo NO`
- Workflow config here: !`ls -d .claude/skills .claude/commands 2>/dev/null || echo "no .claude/skills or .claude/commands"`
- Repo boundary for the context walk: !`test -e .git && echo ".git present" || echo "NO .git — see guard 1"`
- Cluster Redis tunnel on 16379: !`lsof -nP -iTCP:16379 -sTCP:LISTEN >/dev/null 2>&1 && echo "listening" || echo "absent"`

## Your task

Promote the workflow in the current directory into the harness, then explain the result.

`$ARGUMENTS` — the **first** word is the entry prompt name (a file in `.claude/commands/<name>.md`).
Pass any remaining words through to the CLI verbatim. With no arguments, list the available entry
names from `.claude/commands/` and stop; do not guess one.

### Check the three guards first

These are not hypothetical — each one produced a confident, wrong result during development.

1. **No `.git` in this directory.** `promote` bounds its `CLAUDE.md` chain walk at a `.git` entry.
Without one it climbs into ancestor directories and sweeps their context files — including a
personal `~/CLAUDE.md` — into a bundle bound for a shared store. If the Context above says
`NO .git`, say so and offer `git init` before promoting. Do not promote past this silently.

2. **`$SH_HARNESS_DIR` unset or wrong.** The CLI must run from the harness checkout: there is no
installed binary, and the sandbox inventory is resolved relative to the harness _module_, so
running it from here would silently degrade the binary check to `inventory_unavailable`. If it is
unset or the CLI is missing, stop and tell the user to
`export SH_HARNESS_DIR=/path/to/serverless-harness`.

3. **Redis.** Upload must go to the **cluster's** Redis. If the tunnel is `absent`, start it in the
background: `kubectl port-forward -n default svc/redis 16379:6379`. Use **16379**, never 6379 —
this repo's own test container publishes `0.0.0.0:6379`, and promoting into it reports a
successful upload while the harness then fails with `config bundle not found` for that exact
digest. If 16379 is in use by something that is not this tunnel, say so rather than assuming.

### Promote

```bash
HOME="$PWD" REDIS_URL=redis://localhost:16379 \
pnpm --dir "$SH_HARNESS_DIR/harness" promote --entry <entry> --project "$PWD" <extra-args>
```

Two parts of that are load-bearing, so do not "simplify" them:

- **`HOME="$PWD"`** makes this directory its own user scope, so only this workflow travels. Without
it, promote reads your real `~/.claude` — measured at 56 travelling skills and ~8.6 MB versus one
skill and ~12 KB — and it is also what makes this project's `.claude/commands/` entry prompt
visible at all, since promote reads prompts from user scope only.
- **`--project "$PWD"`** is passed explicitly even though it looks redundant: run without it the CLI
promotes whatever directory the process started in, which through `pnpm --dir` is the harness
checkout, not this project.

### Then verify and explain

- If the upload succeeded, confirm the bundle reached the **cluster's** store by reading it back
through the cluster's own client, not through the tunnel — this is the only check that
distinguishes the two Redises:
`kubectl exec -n default deploy/redis -- redis-cli EXISTS "config:bundle:<digest>"` → expect `1`.
- Report: how many skills travelled, how many dropped and **why** for each, whether preflight was
clean, the digest, the bundle size, whether the upload was skipped as unchanged, and the lockfile
path.
- Show the dispatch envelope the CLI prints, with the real digest filled in.

Interpret the exit code rather than echoing the raw error:

- **exit 2 — preflight errors.** Name what blocked. `unknown_entry` means the entry prompt is not in
the bundle; list what is.
- **exit 3 — a structural credential match.** Give the `path:line` and the rule that matched. Say
plainly that this one blocks because a credential reaching a shared store cannot be un-shared by
re-promoting, and that the fix is removing it or adding the file to a deny-list — not a retry.
- **Warnings are not errors.** A clean promotion can still carry warnings (a dropped skill, a
binary the sandbox may not have). Summarise them; do not present them as failure.

Finally, note what promotion does **not** carry: MCP servers and subagents are out of scope, and
promoted memory travels read-only, so a remote run consumes what it was taught locally and returns
discoveries in the leaf result instead of writing back.
134 changes: 134 additions & 0 deletions deploy/claude/tests/promote-command.test.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,134 @@
#!/usr/bin/env bash
# deploy/claude/tests/promote-command.test.sh
#
# Cluster-free tests for the /promote slash command asset. The command is a markdown prompt, so
# nothing type-checks it and nothing runs it in CI -- exactly the conditions under which
# promote-live-smoke.test.ts sat broken for a whole PR. These checks pin the parts that make it
# correct rather than merely present:
#
# - `allowed-tools` must cover every command the body actually invokes -- including after `&&` and
# `||`, which is where `echo` hid while an earlier version of this check claimed to cover
# "every command" and did not -- or the user approves a permission prompt on every run and the
# "one command" claim is false. Grants are matched whole, so tightening `kubectl:*` to per-verb
# grants strengthens the check instead of bypassing it.
# - the two flags whose absence silently promotes the WRONG THING (`--project`, `HOME=`) must be
# in the documented invocation. Both were verified by measurement: without --project the CLI
# promotes the harness checkout it was launched from, and without HOME it resolves 56
# travelling skills and cannot find the entry prompt at all.
# - the redis tunnel must be 16379. On 6379 it reaches this repo's own test container, which
# reports a successful upload and then a `config bundle not found` from the harness.
#
# No cluster required. Run: bash deploy/claude/tests/promote-command.test.sh
set -uo pipefail

CMD="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)/commands/promote.md"
fails=0
check() { if [ "$2" = "$3" ]; then echo " ok: $1"; else
echo " FAIL: $1 (want '$3', got '$2')"
fails=$((fails + 1))
fi; }

echo "== the asset exists and has frontmatter"
check "promote.md exists" "$([ -f "$CMD" ] && echo yes || echo no)" "yes"
[ -f "$CMD" ] || {
echo "FAILED: no command file"
exit 1
}
check "opens with a frontmatter fence" "$(head -1 "$CMD")" "---"
FM="$(awk '/^---$/{n++; next} n==1{print} n>1{exit}' "$CMD")"
for key in description argument-hint allowed-tools; do
check "frontmatter has $key" "$(grep -cE "^$key:" <<< "$FM")" "1"
done

echo "== allowed-tools covers every command the body invokes"
# Grants, kept WHOLE: "Bash(kubectl port-forward:*)" -> "kubectl port-forward". Collapsing these to
# a first word would accept any `kubectl` subcommand against a deliberately per-verb grant, so the
# check would silently weaken exactly when the grant is tightened.
PERMITTED="$(grep -oE 'Bash\([^):]+' <<< "$FM" | sed 's/^Bash(//' | sort -u)"
check "permits pnpm" "$(grep -cE '^pnpm' <<< "$PERMITTED")" "1"
check "grants kubectl per verb, not kubectl:*" \
"$(grep -cxF 'kubectl' <<< "$PERMITTED")" "0"

# Commands the body actually runs. Split on && || ; | and newlines FIRST: keeping only the first
# word of each probe hid every `&& echo`/`|| echo`, which is how `echo` stayed ungranted while a
# test claiming to cover "every command" passed. Env assignments are stripped so
# `HOME=... pnpm ...` is seen as pnpm.
USED="$( {
grep -oE '!`[^`]+`' "$CMD" | sed 's/^!`//; s/`$//'
awk '/^```bash$/{f=1; next} /^```$/{f=0} f' "$CMD"
} | sed -E 's/[[:space:]]*(&&|\|\||;|\|)[[:space:]]*/\n/g' |
sed -E 's/^[[:space:]]+//; s/^[0-9]*>[^[:space:]]*[[:space:]]*//' |
sed -E 's/^([A-Za-z_][A-Za-z0-9_]*=("[^"]*"|[^[:space:]]*)[[:space:]]+)+//' |
grep -oE '^[a-z][a-z0-9_-]*([[:space:]]+[a-z][a-z0-9_-]*)?' | sort -u)"

# Each invocation must be covered by SOME grant, matched as a prefix so a two-word grant
# ("kubectl port-forward") covers a longer invocation and a one-word grant ("ls") still covers "ls -d".
for u in $(printf '%s\n' "$USED" | tr ' ' '@'); do
inv="$(printf '%s' "$u" | tr '@' ' ')"
covered=no
while read -r g; do
[ -z "$g" ] && continue
case "$inv " in "$g "*) covered=yes; break ;; esac
case "$inv" in "$g") covered=yes; break ;; esac
done <<< "$PERMITTED"
# A bare first word of a two-word grant is not itself an invocation (e.g. "kubectl" alone never
# runs); skip it rather than demanding a grant that would have to be widened.
if [ "$covered" = no ] && grep -qE "^$inv " <<< "$PERMITTED"; then covered=skip; fi
case "$covered" in
yes) echo " ok: allowed-tools covers '$inv'" ;;
skip) echo " ok: '$inv' is a grant prefix, not an invocation" ;;
*)
echo " FAIL: '$inv' is invoked but not granted"
fails=$((fails + 1))
;;
esac
done

# kubectl is instructed in prose (guard 3's port-forward, the redis-cli EXISTS verify) rather than
# inside a ! probe or a bash fence, so the extraction above cannot see it. This floor exists to
# cover that blind spot -- it is not redundant with the loop.
for needed in "kubectl port-forward" "kubectl exec"; do
check "grants '$needed', which appears only in prose" \
"$(grep -cxF "$needed" <<< "$PERMITTED")" "1"
done

echo "== the invocation cannot silently promote the wrong thing"
# Counted as present-or-absent: both legitimately appear twice, once in the invocation and once
# in the prose explaining why they are not redundant.
present() { [ "$1" -ge 1 ] && echo 1 || echo 0; }
check "passes --project explicitly" "$(present "$(grep -c -- '--project "\$PWD"' "$CMD")")" "1"
check "sets HOME to the project" "$(present "$(grep -c 'HOME="\$PWD"' "$CMD")")" "1"
check "runs the CLI via --dir on the harness checkout" \
"$(grep -c -- '--dir "\$SH_HARNESS_DIR/harness"' "$CMD")" "1"
check "explains why --project is not redundant" \
"$(grep -ciE 'do not "simplify"|not "simplify" them' "$CMD")" "1"

echo "== the redis tunnel is the private port, not 6379"
check "port-forwards 16379:6379" "$(grep -c '16379:6379' "$CMD")" "1"
check "REDIS_URL uses 16379" "$(grep -c 'redis://localhost:16379' "$CMD")" "1"
check "never points REDIS_URL at 6379" "$(grep -c 'redis://localhost:6379' "$CMD")" "0"
check "says why 6379 is wrong" "$(grep -ciE '0\.0\.0\.0:6379|test container publishes' "$CMD")" "1"

echo "== the three measured guards are all present"
check "guard: .git bounds the context walk" "$(grep -ciE '\.git' "$CMD" | awk '$1>0{print 1; exit} {print 0}')" "1"
check "guard: SH_HARNESS_DIR / inventory resolution" \
"$(grep -ciE 'inventory_unavailable' "$CMD" | awk '$1>0{print 1; exit} {print 0}')" "1"
check "guard: cluster-side verification of the upload" \
"$(grep -c 'redis-cli EXISTS' "$CMD")" "1"

echo "== exit codes are interpreted, not echoed"
check "explains exit 2 (preflight)" "$(grep -c 'exit 2' "$CMD")" "1"
check "explains exit 3 (structural secret)" "$(grep -c 'exit 3' "$CMD")" "1"
check "distinguishes warnings from errors" \
"$(grep -ciE 'Warnings are not errors' "$CMD")" "1"

echo "== it does not overpromise what promotion carries"
check "names MCP and subagents as out of scope" \
"$(grep -ciE 'MCP servers and subagents' "$CMD")" "1"
check "says memory travels read-only" "$(grep -ciE 'read-only' "$CMD" | awk '$1>0{print 1; exit} {print 0}')" "1"

if [ "$fails" -ne 0 ]; then
echo "FAILED: $fails check(s)"
exit 1
fi
echo "PASS"
25 changes: 21 additions & 4 deletions deploy/knative/demo-promoted-workflow.sh
Original file line number Diff line number Diff line change
Expand Up @@ -282,16 +282,33 @@ mapfile -t POOL_PODS < <(kubectl get pods -n "$NS" -l "$POOL_SEL" -o name 2>/dev
exit 1
}
CACHE_DIR="/workspace/.sh-config/sha256-${DIGEST#sha256:}"
# Purge EVERY cached bundle, not just this run's digest. Purging one digest is not enough: any other
# bundle in the cache that happens to carry the same memory fact answers the bare arm just as well.
# Measured -- a bundle built from this same fixture before Prettier normalised one emphasis marker
# in the memory file has a different content digest, sat alongside, and made the control fail with
# the per-digest purge passing.
for p in "${POOL_PODS[@]}"; do
kubectl exec -n "$NS" "$p" -- rm -rf "$CACHE_DIR" 2>/dev/null || true
kubectl exec -n "$NS" "$p" -- sh -c 'rm -rf /workspace/.sh-config/sha256-*' 2>/dev/null || true
done
still=0
unreadable=0
for p in "${POOL_PODS[@]}"; do
kubectl exec -n "$NS" "$p" -- test -e "$CACHE_DIR" 2>/dev/null && still=$((still + 1))
# Distinguish "zero bundles" from "could not look". An empty $n means the exec failed -- pod not
# ready, evicted, transient API error -- and `${n:-0}` would silently score that as a clean pod,
# letting the run claim the cache is empty having verified nothing. That is the same shape of
# false green this whole purge exists to prevent, so it must not be counted as evidence.
n=$(kubectl exec -n "$NS" "$p" -- sh -c 'ls -1d /workspace/.sh-config/sha256-* 2>/dev/null | wc -l' 2>/dev/null | tr -d ' \r')
if [ -z "$n" ]; then
unreadable=$((unreadable + 1))
continue
fi
still=$((still + n))
done
[ "$unreadable" -eq 0 ] ||
ko "could not inspect the cache on $unreadable pod(s); the control is unverified"
[ "$still" -eq 0 ] &&
ok "shared config cache purged from ${#POOL_PODS[@]} pool sandbox(es), so the control is honest" ||
ko "the digest is still cached in $still sandbox(es); the bare run could read it"
ok "config cache emptied on ${#POOL_PODS[@]} pool sandbox(es), so the control is honest" ||
ko "$still cached bundle(s) remain; the bare run could read one of them"

SID_A="demo-bare-$$"
SID_B="demo-promoted-$$"
Expand Down
Loading
Loading