From 08b4d2eef3323a720834603a6663286dd98e3e6c Mon Sep 17 00:00:00 2001 From: Paolo Dettori Date: Wed, 2 Sep 2026 09:05:40 -0400 Subject: [PATCH 1/3] fix(pre-commit): run the hooks again -- 'typescript' is not an identify tag `pre-commit run` executed zero hooks: the config failed validation outright, so shellcheck, gitleaks, check-yaml and the whitespace fixers never ran either. The cause was `types_or: [javascript, typescript, ...]` on the prettier hook -- identify has no `typescript` tag (the tags are `ts` and `tsx`), so `.ts` files were never matched even before newer pre-commit turned the unknown tag into a hard InvalidConfigError. Prettier was also unrunnable outside the hook: `.prettierrc` and `.prettierignore` were committed but no `package.json` declared prettier, so `make fmt` died with ERR_PNPM_RECURSIVE_EXEC_NO_PACKAGE. With the config loading again, two more hooks turned out to be broken -- both invisible for as long as nothing ran them: - hadolint never linted anything. `entry` was overridden to a bare `hadolint`, which is the entry of upstream's *system* hook, so `language: docker_image` resolved an image literally named `hadolint` and every run failed before reaching a Dockerfile. All five Dockerfiles are in fact clean. - shellcheck found a real SC2034 in remote-worker/deploy-incluster.sh, outside the `deploy/` tree that the security-scans.yml shellcheck job covers. Changes: - Add a root `package.json` pinning prettier 3.9.6 exactly, so `make fmt` works from a clean `pnpm install`. - Drive the hook from that same pinned binary via `repo: local` rather than pre-commit/mirrors-prettier, which is archived upstream and was pinned to a Prettier 4 pre-release -- two Prettiers that could format differently. One binary for both means `make fmt` and the hook cannot disagree. - Add `ts`/`tsx` (and `jsx`) so TypeScript is actually covered. - Restore hadolint's image reference, pinned to match `rev` (upstream's own entry is untagged, i.e. :latest). - Set shellcheck to `-S warning`, matching the severity security-scans.yml already enforces, so the hook and that job cannot disagree. The scripts are clean at warning+ once the SC2034 above is fixed; the remaining info/style findings include SC1091 false positives from relative `source` paths and are left to a follow-up. - Add a CI `lint` job running `make lint`, since nothing in CI ran the hooks -- which is why a config running zero hooks looked exactly like a green build. - Document `pre-commit install` in CONTRIBUTING.md. `pre-commit run --all-files` now exits 0 with all nine hooks executing. The repo-wide reformat this unblocks lands in the next commit, kept separate so it does not bury this change. Fixes #196 Assisted-By: Claude (Anthropic AI) Signed-off-by: Paolo Dettori --- .github/workflows/ci.yml | 59 +++++++++++++++++++++++++------ .pre-commit-config.yaml | 30 ++++++++++++---- .prettierignore | 6 ++++ CONTRIBUTING.md | 18 ++++++++-- package.json | 12 +++++++ pnpm-lock.yaml | 13 +++++++ remote-worker/deploy-incluster.sh | 2 +- 7 files changed, 119 insertions(+), 21 deletions(-) create mode 100644 package.json diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 559aa94..77a7736 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -24,27 +24,27 @@ jobs: --health-timeout 5s --health-retries 5 steps: - - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 + - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 with: submodules: recursive - - uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v6 + - uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v6 with: - node-version: "22" + node-version: '22' - - uses: pnpm/action-setup@0977fd99725f1db4007ccb2928dbb4e90d06cc86 # v6.0.10 + - uses: pnpm/action-setup@0977fd99725f1db4007ccb2928dbb4e90d06cc86 # v6.0.10 with: version: 9 - name: Cache pnpm store - uses: actions/cache@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0 + uses: actions/cache@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0 with: path: ~/.local/share/pnpm/store/v3 key: pnpm-${{ hashFiles('pnpm-lock.yaml') }} restore-keys: pnpm- - name: Cache pi-fork node_modules - uses: actions/cache@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0 + uses: actions/cache@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0 with: path: pi-fork/node_modules key: pi-fork-${{ hashFiles('pi-fork/package-lock.json') }} @@ -68,6 +68,43 @@ jobs: - name: Test run: pnpm -r test + # Runs the same `make lint` (pre-commit over all files) that contributors run locally, so + # a config that silently executes zero hooks can no longer look identical to a green + # build -- which is exactly how the broken `typescript` identify tag survived unnoticed. + # Needs node/pnpm because the prettier hook drives the repo's own pinned Prettier, and + # needs the pi-fork submodule because the workspace links into it. + lint: + runs-on: ubuntu-latest + timeout-minutes: 10 + steps: + - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 + with: + submodules: recursive + + - uses: actions/setup-node@820762786026740c76f36085b0efc47a31fe5020 # v6 + with: + node-version: '22' + + - uses: pnpm/action-setup@0977fd99725f1db4007ccb2928dbb4e90d06cc86 # v6.0.10 + with: + version: 9 + + - name: Cache pre-commit environments + uses: actions/cache@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0 + with: + path: ~/.cache/pre-commit + key: pre-commit-${{ hashFiles('.pre-commit-config.yaml') }} + restore-keys: pre-commit- + + - name: Install workspace + run: pnpm install --frozen-lockfile + + - name: Install pre-commit + run: pipx install pre-commit==4.3.0 + + - name: Run hooks + run: make lint + # The deploy/ shell tests are cluster-free (kubectl/kind/docker are mocked on PATH), so # they need no node, no pnpm and no cluster -- hence their own fast job rather than a step # tacked onto `check`. Without this job nothing runs them: security-scans.yml only @@ -76,7 +113,7 @@ jobs: runs-on: ubuntu-latest timeout-minutes: 5 steps: - - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 + - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 - name: Run deploy shell tests run: make test-deploy @@ -85,14 +122,14 @@ jobs: runs-on: ubuntu-latest timeout-minutes: 10 steps: - - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 + - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 - - uses: bufbuild/buf-setup-action@a47c93e0b1648d5651a065437926377d060baa99 # v1 + - uses: bufbuild/buf-setup-action@a47c93e0b1648d5651a065437926377d060baa99 # v1 with: - version: "1.71.0" + version: '1.71.0' github_token: ${{ github.token }} - - uses: actions/setup-go@b7ad1dad31e06c5925ef5d2fc7ad053ef454303e # v7.0.0 + - uses: actions/setup-go@b7ad1dad31e06c5925ef5d2fc7ad053ef454303e # v7.0.0 with: go-version-file: gen/go/go.mod diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml index 7eb78a6..686041e 100644 --- a/.pre-commit-config.yaml +++ b/.pre-commit-config.yaml @@ -12,27 +12,43 @@ repos: args: [--maxkb=500] - id: check-merge-conflict - - repo: https://github.com/pre-commit/mirrors-prettier - rev: v4.0.0-alpha.8 + # Prettier runs from the repo's own pinned devDependency (package.json) instead of + # pre-commit/mirrors-prettier: that mirror is archived upstream and was pinned to a + # Prettier 4 pre-release, so it could format differently from `make fmt`. Driving the + # same binary from both makes them identical by construction. Needs `pnpm install`. + # + # `ts`/`tsx` are the identify tags for TypeScript -- there is no `typescript` tag, and + # naming one silently matched no .ts file and then failed config validation outright. + - repo: local hooks: - id: prettier - types_or: [javascript, typescript, json, yaml, markdown] + name: prettier + entry: pnpm exec prettier --write --ignore-unknown + language: system + types_or: [javascript, jsx, ts, tsx, json, yaml, markdown] exclude: ^(pi-fork/|packages/k8s-sandbox/src/gen/|gen/) + # `-S warning` matches the shellcheck gate in .github/workflows/security-scans.yml + # exactly, so the hook and that job cannot disagree about what fails. The scripts are + # clean at warning+; the remaining info/style findings include SC1091 false positives + # (shellcheck runs from the repo root and cannot resolve `source ./lib.sh`). Tightening + # the severity is a separate change from getting the hooks running at all. - repo: https://github.com/shellcheck-py/shellcheck-py rev: v0.10.0.1 hooks: - id: shellcheck - args: [-x] + args: [-x, -S, warning] files: \.sh$ + # `entry` must stay ` hadolint`: this previously overrode it to a bare `hadolint`, + # which is the entry of upstream's *system* hook, so docker_image resolved an image + # literally named `hadolint` and every run failed before linting anything. The image is + # pinned to match `rev` -- upstream's own entry is untagged, i.e. :latest. - repo: https://github.com/hadolint/hadolint rev: v2.12.0 hooks: - id: hadolint-docker - entry: hadolint - language: docker_image - types: [dockerfile] + entry: ghcr.io/hadolint/hadolint:v2.12.0 hadolint - repo: https://github.com/gitleaks/gitleaks rev: v8.21.2 diff --git a/.prettierignore b/.prettierignore index 77111b4..4079333 100644 --- a/.prettierignore +++ b/.prettierignore @@ -5,3 +5,9 @@ dist/ pnpm-lock.yaml packages/k8s-sandbox/src/gen/ gen/ + +# Sibling checkouts of this same repo. `prettier --write .` (make fmt) walks the directory tree +# and reads only this file -- not .gitignore -- so without these it reformats files belonging to +# other worktrees, dirtying branches that have nothing to do with the current one. +.worktrees/ +.claude/worktrees/ diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index b55f458..2a6dea8 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -7,7 +7,8 @@ - Node.js 22+ - pnpm 9+ - Redis (for work-queue tests) -- Docker (for container builds) +- Docker (for container builds, and for the hadolint pre-commit hook) +- [pre-commit](https://pre-commit.com/#install) (git hooks) ### Getting Started @@ -22,6 +23,9 @@ cd pi-fork && npm ci && npm run build && cd .. # Install workspace dependencies pnpm install +# Install the git hooks (formatting, shellcheck, secret scanning) +pre-commit install + # Run tests pnpm -r test @@ -29,6 +33,11 @@ pnpm -r test cd harness && pnpm exec tsc --noEmit ``` +`pre-commit install` is not optional in practice: the same hooks run in CI as the `lint` +job, so skipping it just moves the failure later. Run them over the whole tree at any time +with `make lint`, and format without linting with `make fmt`. Both drive the single Prettier +version pinned in the root `package.json`, so local and CI formatting cannot diverge. + ### Workspace Structure ``` @@ -52,18 +61,23 @@ serverless-harness/ 3. Make your changes with tests 4. Ensure `pnpm -r test` passes 5. Ensure `tsc --noEmit` passes in all packages with tsconfigs -6. Submit a pull request +6. Ensure `make lint` passes (or commit with the hooks installed, which is equivalent) +7. Submit a pull request ### CI Checks All PRs must pass: + - **Typecheck**: `tsc --noEmit` across harness, k8s-sandbox, knative-server, experiments - **Tests**: `pnpm -r test` (requires Redis for work-queue) +- **Lint**: `make lint` -- every pre-commit hook over all files (Prettier, shellcheck, + hadolint, gitleaks, YAML and whitespace checks) - **DCO**: All commits must be signed off ## Commit Messages Use conventional commit format: + - `feat:` New features - `fix:` Bug fixes - `docs:` Documentation changes diff --git a/package.json b/package.json new file mode 100644 index 0000000..ed7e236 --- /dev/null +++ b/package.json @@ -0,0 +1,12 @@ +{ + "name": "serverless-harness", + "private": true, + "description": "Workspace root. Holds only repo-wide tooling -- runtime code lives in harness/, packages/* and experiments/.", + "scripts": { + "fmt": "prettier --write .", + "fmt:check": "prettier --check ." + }, + "devDependencies": { + "prettier": "3.9.6" + } +} diff --git a/pnpm-lock.yaml b/pnpm-lock.yaml index bde4e24..c43e58f 100644 --- a/pnpm-lock.yaml +++ b/pnpm-lock.yaml @@ -6,6 +6,12 @@ settings: importers: + .: + devDependencies: + prettier: + specifier: 3.9.6 + version: 3.9.6 + experiments: dependencies: '@earendil-works/pi-ai': @@ -844,6 +850,11 @@ packages: resolution: {integrity: sha512-FfR8sjd4em2T6fb3I2MwAJU7HWVMr9zba+enmQeeWFfCbm+UOC/0X4DS8XtpUTMwWMGbjKYP7xjfNekzyGmB3A==} engines: {node: ^10 || ^12 || >=14} + prettier@3.9.6: + resolution: {integrity: sha512-OpN0zzVdiaiAhxpuuj5efpIS4sY9j7bY6uR5mnj5yPzGkdkjNKSJeUThPb60Jw29QuAZgA4o+/iB49kFiaBX6g==} + engines: {node: '>=14'} + hasBin: true + protobufjs@7.6.5: resolution: {integrity: sha512-/FPD0nUc9jH6rfFjji9IBqOz4pcSE3CsT1m7Ep6Mdb0LxSUMj8hgl6GomOvZzpNpAqqGaXA0P3VSrZLFzIhQrw==} engines: {node: '>=12.0.0'} @@ -1468,6 +1479,8 @@ snapshots: picocolors: 1.1.1 source-map-js: 1.2.1 + prettier@3.9.6: {} + protobufjs@7.6.5: dependencies: '@protobufjs/aspromise': 1.1.2 diff --git a/remote-worker/deploy-incluster.sh b/remote-worker/deploy-incluster.sh index f35360f..a669d6b 100755 --- a/remote-worker/deploy-incluster.sh +++ b/remote-worker/deploy-incluster.sh @@ -29,7 +29,7 @@ sed -e "s#__IMAGE__#${IMAGE}#g" -e "s#__SANDBOX_ID__#${SANDBOX_ID}#g" \ oc rollout status deploy/remote-worker -n "$NS" --timeout=120s echo "==> presence in Redis (worker registered via its live Attach stream)" -for i in $(seq 1 20); do +for _ in $(seq 1 20); do rec="$(oc exec deploy/redis -n "$NS" -- redis-cli HGET sh:sandbox:records "$SANDBOX_ID" 2>/dev/null || true)" [ -n "$rec" ] && { echo "$rec"; break; } sleep 1 From b6f85a4d6bf2c813dd98cc49ca8ca94a60f596b1 Mon Sep 17 00:00:00 2001 From: Paolo Dettori Date: Wed, 2 Sep 2026 11:26:14 -0400 Subject: [PATCH 2/3] style: apply prettier across the tree for the first time Mechanical output of `make fmt`, no hand edits. This is the backlog from the previous commit: the prettier hook aborted config validation, and its `types_or` never named a real TypeScript tag anyway, so `.ts` files were never formatted even when the config loaded. Kept as its own commit so the fix that unblocks it stays reviewable. Mostly quote style (`.prettierrc` sets singleQuote), trailing commas, comment alignment, markdown table padding and YAML flow-sequence reflow. Verified semantics-preserving: - all 31 changed YAML/JSON files parse to documents identical to their previous contents (compared as parsed structures, not text) - `make typecheck` clean - `pnpm -r test`: 851 passed, 15 skipped - `make test-deploy` passes - `pre-commit run --all-files` exits 0 with all nine hooks running Regenerated rather than replayed when rebasing onto main after #203 and #204, so it also covers the code #204 added -- replaying the old diff would have conflicted with it on run-leaf.ts for no benefit, formatting being mechanical. Assisted-By: Claude (Anthropic AI) Signed-off-by: Paolo Dettori --- .github/dependabot.yml | 2 +- .github/workflows/build.yaml | 12 +- .github/workflows/scorecard.yml | 10 +- .github/workflows/security-scans.yml | 20 +- .hadolint.yaml | 2 +- CLAUDE.md | 16 +- README.md | 67 +-- SECURITY.md | 2 +- deploy/knative/EXPERIMENTS.md | 61 +- deploy/knative/README-authbridge.md | 48 +- deploy/knative/README-k8s.md | 36 +- deploy/knative/README-kind.md | 74 +-- deploy/knative/README-ocp.md | 62 +- deploy/knative/README-worker.md | 53 +- deploy/knative/SMOKE.md | 20 +- deploy/knative/authbridge/ab1-deployment.yaml | 8 +- deploy/knative/echo-target.yaml | 8 +- deploy/knative/echo-target/echo.js | 27 +- deploy/knative/gitd.yaml | 8 +- deploy/knative/ibac-stub.yaml | 16 +- deploy/knative/leaf-cron.yaml | 14 +- deploy/knative/leaf-orchestrator.yaml | 2 +- deploy/knative/leaf-scaledjob.yaml | 30 +- .../overlays/ocp-authbridge/patch-ab1.yaml | 2 +- .../ocp-authbridge/patch-echo-target.yaml | 2 +- .../ocp-authbridge/patch-ibac-stub.yaml | 2 +- .../ocp-authbridge/patch-sandbox-ab2.yaml | 4 +- .../knative/overlays/ocp/patch-sandbox.yaml | 4 +- deploy/knative/redis.yaml | 6 +- deploy/knative/relay-deployment.yaml | 10 +- deploy/knative/sandbox-pool-ab2.yaml | 96 +-- deploy/knative/sandbox-pool.yaml | 52 +- deploy/knative/sandbox.yaml | 15 +- deploy/knative/service.yaml | 46 +- deploy/knative/swebench-sandbox-pool.yaml | 42 +- deploy/knative/worker-example.yaml | 2 +- docs/adrs/0000-adr-template.md | 6 +- docs/adrs/0001-redis-session-backend.md | 4 +- .../0002-k8s-sandbox-client-remote-exec.md | 2 +- docs/adrs/0003-persistent-in-pod-channel.md | 2 +- docs/adrs/0004-knative-serverless-wrapper.md | 2 +- docs/adrs/0005-mcp-code-mode.md | 4 +- .../0006-generalized-credentialed-egress.md | 2 +- .../0007-compaction-checkpoint-fast-path.md | 2 +- docs/adrs/0008-experiments-harness.md | 2 +- docs/adrs/0009-cluster-experiments.md | 2 +- docs/adrs/0010-identity-spine.md | 2 +- docs/adrs/0011-harness-lockdown.md | 2 +- docs/adrs/0012-inference-injector.md | 2 +- ...3-leaf-session-backend-reprioritization.md | 2 +- docs/adrs/0014-mvp-leaf-session-contract.md | 2 +- docs/adrs/0015-async-leaf-completion.md | 2 +- docs/adrs/0016-human-gate.md | 2 +- ...0017-registry-securitycontext-hardening.md | 2 +- docs/adrs/0018-scheduled-leaf-dispatch.md | 2 +- docs/adrs/0019-ocp-fs-free-deployment.md | 2 +- docs/adrs/0020-fs-free-harness.md | 2 +- docs/adrs/0021-shared-sandbox-pool.md | 2 +- ...022-workload-parameterized-sandbox-load.md | 2 +- .../0023-sandbox-sharing-ratio-experiments.md | 2 +- .../0024-sandbox-transport-remote-exec.md | 6 +- .../0025-authbridge-deployment-topology.md | 18 +- docs/adrs/0026-rc1-static-inject-plugin.md | 4 +- ...7-rc1-control-gate-and-hop2-realization.md | 2 +- docs/adrs/0028-async-prompt-dispatch.md | 4 +- docs/adrs/0029-turn-sse-streaming.md | 4 +- docs/adrs/README.md | 80 +-- docs/demos/README.md | 26 +- docs/demos/remote-sandbox-demo.md | 80 +-- docs/demos/serverless-harness-demo.md | 20 +- docs/executive-overview-leaf-session.md | 65 +- docs/experiment-results.md | 39 +- docs/notes/swebench-image-facts.md | 74 +-- docs/plans/README.md | 12 +- ...6-06-16-m1-redis-session-backend-design.md | 61 +- ...2026-06-17-m2-k8s-sandbox-client-design.md | 72 +-- ...2026-06-17-m3-persistent-channel-design.md | 116 ++-- ...17-m4-knative-serverless-wrapper-design.md | 112 ++-- .../2026-06-18-m10-mcp-code-mode-design.md | 134 ++--- ...-generalized-credentialed-egress-design.md | 67 ++- ...6-06-23-m5-compaction-checkpoint-design.md | 135 +++-- .../specs/2026-06-24-m6-experiments-design.md | 98 +-- ...026-06-25-m7-cluster-experiments-design.md | 81 +-- .../2026-06-26-harness-lockdown-design.md | 104 ++-- .../specs/2026-06-26-identity-spine-design.md | 78 +-- .../2026-06-26-inference-injector-design.md | 60 +- ...leaf-session-backend-capability-charter.md | 106 ++-- ...-06-26-mvp-leaf-session-contract-design.md | 45 +- ...-06-26-pipeline-archetypes-requirements.md | 174 +++--- ...2026-06-27-async-leaf-completion-design.md | 62 +- docs/specs/2026-06-28-human-gate-design.md | 98 +-- ...06-28-registry-hardening-hygiene-design.md | 17 +- ...26-06-28-scheduled-leaf-dispatch-design.md | 59 +- ...2-p0prime-ocp-fs-free-deployment-design.md | 46 +- .../2026-07-02-p1-fs-free-harness-design.md | 51 +- ...026-07-02-p2-shared-sandbox-pool-design.md | 66 +- ...kload-parameterized-sandbox-load-design.md | 32 +- ...andbox-sharing-ratio-experiments-design.md | 52 +- ...026-07-08-sandbox-transport-grpc-design.md | 120 ++-- ...hbridge-egress-control-plane-poc-design.md | 78 +-- ...20-multi-protocol-model-provider-design.md | 42 +- ...2026-08-25-async-prompt-dispatch-design.md | 72 ++- ...26-08-26-st4-go-reference-worker-design.md | 100 +-- .../2026-08-26-turn-sse-streaming-design.md | 114 ++-- ...08-30-seam-output-cap-truncation-design.md | 67 ++- docs/specs/README.md | 102 ++-- experiments/README.md | 12 +- experiments/RESULTS.md | 12 +- experiments/src/counting-backend.ts | 2 +- experiments/src/report.ts | 14 +- experiments/src/session-fixture.ts | 12 +- experiments/src/sharing.ts | 21 +- experiments/src/workload.ts | 97 ++- experiments/swebench/RUNBOOK.md | 60 +- experiments/swebench/deck.json | 146 ++--- experiments/test/counting-backend.test.ts | 21 +- .../test/e2-reconstruction-cost.test.ts | 43 +- experiments/test/e5-budget-live.test.ts | 23 +- experiments/test/e5-budget-structural.test.ts | 41 +- .../test/e6-saturation-structural.test.ts | 36 +- .../e7-converge-contention-structural.test.ts | 30 +- experiments/test/predictions.test.ts | 16 +- experiments/test/report.test.ts | 34 +- experiments/test/session-fixture.test.ts | 26 +- experiments/test/sharing-benefit.test.ts | 44 +- experiments/test/swebench-deck.test.ts | 80 ++- .../test/swebench-sandbox-build.test.ts | 134 ++--- experiments/test/workload.test.ts | 58 +- experiments/vitest.config.ts | 4 +- harness/README.md | 5 + harness/src/budget-voter.ts | 32 +- harness/src/buffered-redis-backend.ts | 6 +- harness/src/checkpoint-extension.ts | 13 +- harness/src/classify-outcome.ts | 4 +- harness/src/cli.ts | 2 +- harness/src/converge.ts | 47 +- harness/src/flush-extension.ts | 8 +- harness/src/gate.ts | 93 ++- harness/src/index.ts | 9 +- harness/src/leaf-job-runner.ts | 50 +- harness/src/leaf-result-store.ts | 74 ++- harness/src/pool-records.ts | 8 +- harness/src/request-approval-tool.ts | 51 +- harness/src/run-leaf.ts | 277 ++++++--- harness/src/run-turn.ts | 133 ++-- harness/src/sandbox-lease.ts | 9 +- harness/src/select-sandbox.ts | 32 +- harness/src/submit-verdict-tool.ts | 41 +- harness/src/swebench-setup.ts | 67 ++- harness/src/tool-choice-extension.ts | 12 +- harness/src/turn-stream.ts | 51 +- harness/src/verdict-termination-extension.ts | 15 +- harness/src/verdict.ts | 16 +- harness/test/budget-voter.test.ts | 96 +-- harness/test/buffered-redis-backend.test.ts | 72 ++- harness/test/checkpoint.test.ts | 97 +-- harness/test/classify-outcome.test.ts | 51 +- harness/test/converge.test.ts | 136 +++-- harness/test/fixtures.test.ts | 24 +- harness/test/gate.test.ts | 214 ++++--- harness/test/integration.test.ts | 24 +- harness/test/leaf-job-runner.test.ts | 206 ++++--- harness/test/leaf-result-store.test.ts | 171 ++++-- harness/test/model-gateway.test.ts | 75 +-- harness/test/pool-live-smoke.test.ts | 24 +- harness/test/pool-records.test.ts | 18 +- harness/test/request-approval-tool.test.ts | 67 ++- harness/test/run-leaf.test.ts | 568 +++++++++++------- harness/test/run-turn-model.test.ts | 216 +++---- harness/test/run-turn-sandbox.test.ts | 47 +- harness/test/run-turn.test.ts | 74 +-- harness/test/sandbox-lease.test.ts | 24 +- harness/test/select-sandbox.test.ts | 180 +++--- harness/test/submit-verdict-tool.test.ts | 90 ++- harness/test/swebench-setup.test.ts | 121 ++-- harness/test/tool-choice-extension.test.ts | 50 +- harness/test/turn-stream.test.ts | 135 +++-- harness/test/verdict-recovery.test.ts | 50 +- .../verdict-termination-extension.test.ts | 52 +- harness/test/verdict.test.ts | 29 +- harness/vitest.config.ts | 4 +- packages/ibac-stub/src/decide.ts | 24 +- packages/ibac-stub/src/index.ts | 6 +- packages/ibac-stub/src/main.ts | 14 +- packages/ibac-stub/src/server.ts | 59 +- packages/ibac-stub/test/decide.test.ts | 48 +- packages/ibac-stub/test/main.test.ts | 22 +- packages/ibac-stub/test/server.test.ts | 66 +- packages/ibac-stub/vitest.config.ts | 4 +- packages/k8s-sandbox/NOTES-pi-operations.md | 7 +- packages/k8s-sandbox/README.md | 12 +- packages/k8s-sandbox/SMOKE.md | 12 +- packages/k8s-sandbox/deploy/sandbox.yaml | 4 +- packages/k8s-sandbox/src/config.ts | 4 +- packages/k8s-sandbox/src/exec.ts | 47 +- packages/k8s-sandbox/src/extension.ts | 47 +- packages/k8s-sandbox/src/framing.ts | 10 +- packages/k8s-sandbox/src/grep-tool.ts | 46 +- .../k8s-sandbox/src/grpc-relay-transport.ts | 51 +- packages/k8s-sandbox/src/index.ts | 47 +- packages/k8s-sandbox/src/operations.ts | 31 +- packages/k8s-sandbox/src/paths.ts | 2 +- packages/k8s-sandbox/src/persistent-exec.ts | 50 +- packages/k8s-sandbox/src/pool.ts | 21 +- packages/k8s-sandbox/src/req-id.ts | 2 +- packages/k8s-sandbox/src/resolve-pod.ts | 44 +- packages/k8s-sandbox/src/transport.ts | 2 +- packages/k8s-sandbox/test/config.test.ts | 44 +- packages/k8s-sandbox/test/conformance.ts | 64 +- packages/k8s-sandbox/test/exec.test.ts | 62 +- packages/k8s-sandbox/test/extension.test.ts | 32 +- packages/k8s-sandbox/test/framing.test.ts | 163 ++--- packages/k8s-sandbox/test/grep-tool.test.ts | 64 +- .../test/grpc-relay-transport.test.ts | 73 +-- packages/k8s-sandbox/test/live-relay.test.ts | 75 +-- .../k8s-sandbox/test/m3-live-smoke.test.ts | 145 ++--- packages/k8s-sandbox/test/operations.test.ts | 227 +++---- .../test/output-cap-coupling.test.ts | 20 +- packages/k8s-sandbox/test/paths.test.ts | 28 +- .../test/persistent-exec-conformance.test.ts | 42 +- .../k8s-sandbox/test/persistent-exec.test.ts | 144 +++-- packages/k8s-sandbox/test/pool.test.ts | 55 +- .../k8s-sandbox/test/proto-contract.test.ts | 48 +- packages/k8s-sandbox/test/req-id.test.ts | 19 +- packages/k8s-sandbox/test/resolve-pod.test.ts | 118 +++- .../test/transport-conformance.test.ts | 42 +- .../knative-server/src/context-service.ts | 34 +- packages/knative-server/src/cron-dispatch.ts | 32 +- packages/knative-server/src/index.ts | 2 +- packages/knative-server/src/leaf-job.ts | 41 +- packages/knative-server/src/server.ts | 286 +++++---- .../test/authbridge-manifests.test.ts | 203 ++++--- .../test/context-service.test.ts | 100 +-- .../knative-server/test/cron-dispatch.test.ts | 115 ++-- .../test/harness-egress-policy.test.ts | 16 +- .../test/prompt-envelope.test.ts | 36 +- .../test/relay-deployment.test.ts | 72 ++- .../test/run-leaf-async-route.test.ts | 124 ++-- .../test/run-leaf-route.test.ts | 221 ++++--- packages/knative-server/test/server.test.ts | 291 ++++----- .../test/solve-envelope.test.ts | 50 +- .../test/swebench-sandbox-pool.test.ts | 110 ++-- .../test/worker-deployment.test.ts | 78 +-- .../test/workload-route.test.ts | 156 ++--- packages/knative-server/vitest.config.ts | 4 +- packages/sandbox-relay/package.json | 4 +- packages/sandbox-relay/src/index.ts | 4 +- packages/sandbox-relay/src/main.ts | 35 +- packages/sandbox-relay/src/relay.ts | 30 +- .../test/main-default-token.test.ts | 49 +- .../sandbox-relay/test/main-wiring.test.ts | 77 ++- .../sandbox-relay/test/relay-attach.test.ts | 84 +-- .../sandbox-relay/test/relay-exec.test.ts | 104 +++- .../NOTES-pi-sessionmanager.md | 224 +++---- packages/session-backend/src/backend.ts | 2 +- packages/session-backend/src/entry.ts | 14 +- packages/session-backend/src/index.ts | 8 +- packages/session-backend/src/redis-backend.ts | 24 +- packages/session-backend/test/entry.test.ts | 32 +- .../test/redis-backend.test.ts | 86 +-- packages/work-queue/package.json | 17 +- packages/work-queue/src/index.ts | 4 +- packages/work-queue/src/queue.ts | 93 ++- packages/work-queue/test/queue.test.ts | 85 +-- pnpm-workspace.yaml | 6 +- remote-worker/DESIGN.md | 47 +- remote-worker/worker-deployment.yaml | 8 +- 267 files changed, 8066 insertions(+), 6017 deletions(-) diff --git a/.github/dependabot.yml b/.github/dependabot.yml index a438b0f..495f66c 100644 --- a/.github/dependabot.yml +++ b/.github/dependabot.yml @@ -9,7 +9,7 @@ updates: # bumped together — they fail CI if versions diverge across steps. codeql-action: patterns: - - "github/codeql-action*" + - 'github/codeql-action*' - package-ecosystem: npm directory: / diff --git a/.github/workflows/build.yaml b/.github/workflows/build.yaml index 36fcf1f..b25fdc9 100644 --- a/.github/workflows/build.yaml +++ b/.github/workflows/build.yaml @@ -46,18 +46,18 @@ jobs: file: deploy/knative/echo-target/Dockerfile steps: - name: Checkout - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 + uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 with: submodules: recursive - name: Set up QEMU - uses: docker/setup-qemu-action@96fe6ef7f33517b61c61be40b68a1882f3264fb8 # v4 + uses: docker/setup-qemu-action@96fe6ef7f33517b61c61be40b68a1882f3264fb8 # v4 - name: Set up Docker Buildx - uses: docker/setup-buildx-action@37fe631027851001ddb9b187196cc803df7f5f0e # v4 + uses: docker/setup-buildx-action@37fe631027851001ddb9b187196cc803df7f5f0e # v4 - name: Log in to ghcr.io - uses: docker/login-action@dbcb813823bdd20940b903addbd779551569679f # v4 + uses: docker/login-action@dbcb813823bdd20940b903addbd779551569679f # v4 with: registry: ${{ env.REGISTRY }} username: ${{ github.actor }} @@ -65,7 +65,7 @@ jobs: - name: Extract Docker metadata id: meta - uses: docker/metadata-action@dc802804100637a589fabce1cb79ff13a1411302 # v6 + uses: docker/metadata-action@dc802804100637a589fabce1cb79ff13a1411302 # v6 with: images: ${{ matrix.image }} tags: | @@ -74,7 +74,7 @@ jobs: type=semver,pattern={{version}} - name: Build and push - uses: docker/build-push-action@53b7df96c91f9c12dcc8a07bcb9ccacbed38856a # v7 + uses: docker/build-push-action@53b7df96c91f9c12dcc8a07bcb9ccacbed38856a # v7 with: context: ${{ matrix.context }} file: ${{ matrix.file }} diff --git a/.github/workflows/scorecard.yml b/.github/workflows/scorecard.yml index 9f04074..afbf1bd 100644 --- a/.github/workflows/scorecard.yml +++ b/.github/workflows/scorecard.yml @@ -4,7 +4,7 @@ on: push: branches: [main] schedule: - - cron: "30 6 * * 1" + - cron: '30 6 * * 1' workflow_dispatch: permissions: read-all @@ -16,22 +16,22 @@ jobs: security-events: write id-token: write steps: - - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 + - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 with: persist-credentials: false - - uses: ossf/scorecard-action@2d1146689b8cda280b9bc96326124645441f03bc # v2.4.4 + - uses: ossf/scorecard-action@2d1146689b8cda280b9bc96326124645441f03bc # v2.4.4 with: results_file: results.sarif results_format: sarif publish_results: true - - uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1 + - uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1 with: name: scorecard-results path: results.sarif retention-days: 30 - - uses: github/codeql-action/upload-sarif@db488ddef3bf6cb639b32c2e9a7c0a7ea8271d28 # v4 + - uses: github/codeql-action/upload-sarif@db488ddef3bf6cb639b32c2e9a7c0a7ea8271d28 # v4 with: sarif_file: results.sarif diff --git a/.github/workflows/security-scans.yml b/.github/workflows/security-scans.yml index 9576590..03e4c11 100644 --- a/.github/workflows/security-scans.yml +++ b/.github/workflows/security-scans.yml @@ -13,8 +13,8 @@ jobs: contents: read pull-requests: write steps: - - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 - - uses: actions/dependency-review-action@a1d282b36b6f3519aa1f3fc636f609c47dddb294 # v4 + - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 + - uses: actions/dependency-review-action@a1d282b36b6f3519aa1f3fc636f609c47dddb294 # v4 with: fail-on-severity: high deny-licenses: GPL-3.0, AGPL-3.0 @@ -26,8 +26,8 @@ jobs: contents: read security-events: write steps: - - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 - - uses: aquasecurity/trivy-action@ed142fd0673e97e23eac54620cfb913e5ce36c25 # v0.36.0 + - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 + - uses: aquasecurity/trivy-action@ed142fd0673e97e23eac54620cfb913e5ce36c25 # v0.36.0 with: scan-type: fs scan-ref: . @@ -35,7 +35,7 @@ jobs: exit-code: 1 format: sarif output: trivy-results.sarif - - uses: github/codeql-action/upload-sarif@db488ddef3bf6cb639b32c2e9a7c0a7ea8271d28 # v4 + - uses: github/codeql-action/upload-sarif@db488ddef3bf6cb639b32c2e9a7c0a7ea8271d28 # v4 if: always() with: sarif_file: trivy-results.sarif @@ -49,19 +49,19 @@ jobs: matrix: language: [javascript-typescript] steps: - - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 - - uses: github/codeql-action/init@db488ddef3bf6cb639b32c2e9a7c0a7ea8271d28 # v4 + - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 + - uses: github/codeql-action/init@db488ddef3bf6cb639b32c2e9a7c0a7ea8271d28 # v4 with: languages: ${{ matrix.language }} queries: security-extended - - uses: github/codeql-action/analyze@db488ddef3bf6cb639b32c2e9a7c0a7ea8271d28 # v4 + - uses: github/codeql-action/analyze@db488ddef3bf6cb639b32c2e9a7c0a7ea8271d28 # v4 shellcheck: runs-on: ubuntu-latest permissions: contents: read steps: - - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 + - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 - name: Run shellcheck run: find deploy/ -name '*.sh' -exec shellcheck -x -S warning {} + @@ -70,6 +70,6 @@ jobs: permissions: contents: read steps: - - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 + - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1 - name: Run hadolint run: docker run --rm -i -v "$PWD/.hadolint.yaml:/.config/hadolint.yaml" hadolint/hadolint < Dockerfile diff --git a/.hadolint.yaml b/.hadolint.yaml index 9563b2d..a326c72 100644 --- a/.hadolint.yaml +++ b/.hadolint.yaml @@ -1,2 +1,2 @@ ignored: - - DL3018 # Alpine package version pinning breaks on minor bumps + - DL3018 # Alpine package version pinning breaks on minor bumps diff --git a/CLAUDE.md b/CLAUDE.md index 22b720e..3117b07 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -24,15 +24,15 @@ serverless-harness/ ## Key Commands -| Task | Command | -|------|---------| -| Install deps | `pnpm install` | +| Task | Command | +| ------------------- | --------------------------------------- | +| Install deps | `pnpm install` | | Build pi-fork types | `cd pi-fork && npm ci && npm run build` | -| Lint | `make lint` | -| Format | `make fmt` | -| Test (all) | `make test` | -| Typecheck | `make typecheck` | -| Pre-commit install | `pre-commit install` | +| Lint | `make lint` | +| Format | `make fmt` | +| Test (all) | `make test` | +| Typecheck | `make typecheck` | +| Pre-commit install | `pre-commit install` | ## Development Setup diff --git a/README.md b/README.md index 8f01797..afa1977 100644 --- a/README.md +++ b/README.md @@ -2,7 +2,7 @@ **Run stateful AI coding agents serverless — scale to zero between turns, resume exactly where they left off.** -![status](https://img.shields.io/badge/status-MVP%20(Phase%201)-success) +![status]() ![platform](https://img.shields.io/badge/platform-Knative%20%2B%20KEDA-blue) ![runtime](https://img.shields.io/badge/runtime-Pi%20coding%20agent-informational) ![node](https://img.shields.io/badge/node-22%2B-green) @@ -40,13 +40,13 @@ rest. ## Why -| Persistent agent | Serverless Harness | -|------------------|--------------------| -| Process stays resident between turns | Scales to **zero** when idle, cold-starts in sub-second | +| Persistent agent | Serverless Harness | +| --------------------------------------------------- | --------------------------------------------------------------------- | +| Process stays resident between turns | Scales to **zero** when idle, cold-starts in sub-second | | State lives in process memory — lost on crash/evict | State lives in **Redis** — survives eviction, restart, and cold start | -| Tools execute in the agent process | Tools execute in an **isolated sandbox pod** (brain/hands split) | -| Idle compute billed continuously | **Only Redis + sandbox** stay resident (2 pods at rest) | -| One invocation model | **Four**: sync, async fan-out, scheduled, human-gated | +| Tools execute in the agent process | Tools execute in an **isolated sandbox pod** (brain/hands split) | +| Idle compute billed continuously | **Only Redis + sandbox** stay resident (2 pods at rest) | +| One invocation model | **Four**: sync, async fan-out, scheduled, human-gated | In an idle-heavy workload [experiment](deploy/knative/EXPERIMENTS.md), the serverless path consumed roughly **a quarter** of the pod-seconds of an equivalent always-on agent — because the expensive @@ -68,14 +68,14 @@ flowchart LR R <-->|session state| Q ``` -| Component | Role | -|-----------|------| -| **Knative Service** | Scale-to-zero HTTP endpoint; runs a turn inline (sync) or enqueues it (async) | -| **Redis** | Durable session state (resume by `sessionId`), work queue (Streams), gate state | -| **KEDA ScaledJob** | Autoscales `leaf-worker` pods 0→N on queue depth (`lagCount` + `pendingEntriesCount`) | -| **sandbox-0** | Persistent pod where all tool/code execution runs; reached via `kubectl exec` | -| **Shared PVC** | Volume-envelope contract — inputs, results, and markers travel as files | -| **CronJob** | Scheduled dispatch (`cron-dispatch`) for periodic batch work | +| Component | Role | +| ------------------- | ------------------------------------------------------------------------------------- | +| **Knative Service** | Scale-to-zero HTTP endpoint; runs a turn inline (sync) or enqueues it (async) | +| **Redis** | Durable session state (resume by `sessionId`), work queue (Streams), gate state | +| **KEDA ScaledJob** | Autoscales `leaf-worker` pods 0→N on queue depth (`lagCount` + `pendingEntriesCount`) | +| **sandbox-0** | Persistent pod where all tool/code execution runs; reached via `kubectl exec` | +| **Shared PVC** | Volume-envelope contract — inputs, results, and markers travel as files | +| **CronJob** | Scheduled dispatch (`cron-dispatch`) for periodic batch work | > **Note:** The Knative Service and the `leaf-worker` are the **same container image** with two entry > points (`server.ts` vs `leaf-job.ts`). Both converge on `runLeaf()`, which routes execution into @@ -182,11 +182,11 @@ troubleshooting — is in **[`deploy/knative/README-ocp.md`](deploy/knative/READ The same backend serves three orchestration patterns, all validated end-to-end on Kind: -| Archetype | Pattern | Example use case | -|-----------|---------|------------------| -| **A — Async fan-out** | `{async:true}` → Redis Streams → KEDA scales workers 0→N → done-markers | "Research 10 topics concurrently" | -| **B — Human gate** | Leaf pauses → `awaiting_approval` → external verdict → resume/terminate | "Draft a clause, pause for legal sign-off, finalize" | -| **C — Scheduled** | CronJob → `cron-dispatch` reads a config list → posts each as async | "Summarize yesterday's tickets at 02:00 daily" | +| Archetype | Pattern | Example use case | +| --------------------- | ----------------------------------------------------------------------- | ---------------------------------------------------- | +| **A — Async fan-out** | `{async:true}` → Redis Streams → KEDA scales workers 0→N → done-markers | "Research 10 topics concurrently" | +| **B — Human gate** | Leaf pauses → `awaiting_approval` → external verdict → resume/terminate | "Draft a clause, pause for legal sign-off, finalize" | +| **C — Scheduled** | CronJob → `cron-dispatch` reads a config list → posts each as async | "Summarize yesterday's tickets at 02:00 daily" | --- @@ -228,17 +228,17 @@ backend with all three dispatch archetypes. See the [milestone registry](docs/specs/README.md) for the source-of-truth status of every milestone. **Phase 2 — Zero-Trust Credential Plane (design complete, deferred):** a credential plane where -*no component influenced by model output ever holds a raw secret.* - -| ID | Adds | -|----|------| -| Z1 | Per-session SPIFFE identity (SPIRE) | -| Z2 | Secret-free, default-deny harness lock-down | -| Z3 | Inference injector — provider-key chokepoint, mTLS to the LLM gateway | -| Z4 | MCP code-mode in the sandbox | -| Z5 | Generalized credentialed egress (sandbox forward proxy) | -| Z6 | Subagents as isolated child sessions | -| Z7 | Red-team + formal validation of the credential plane | +_no component influenced by model output ever holds a raw secret._ + +| ID | Adds | +| --- | --------------------------------------------------------------------- | +| Z1 | Per-session SPIFFE identity (SPIRE) | +| Z2 | Secret-free, default-deny harness lock-down | +| Z3 | Inference injector — provider-key chokepoint, mTLS to the LLM gateway | +| Z4 | MCP code-mode in the sandbox | +| Z5 | Generalized credentialed egress (sandbox forward proxy) | +| Z6 | Subagents as isolated child sessions | +| Z7 | Red-team + formal validation of the credential plane | Today the harness uses a trust-the-operator model: the model credential is a pre-provisioned Kubernetes Secret, there is no egress policy, and all leaves share one service-account identity. Those @@ -258,9 +258,8 @@ gaps are exactly what Phase 2 closes. ## Status & License -This is explorative work. It is an MVP — the scale-to-zero, durable-resume, sandbox-isolation, -and dispatch features above are built and smoke-verified; the zero-trust credential plane is +This is explorative work. It is an MVP — the scale-to-zero, durable-resume, sandbox-isolation, +and dispatch features above are built and smoke-verified; the zero-trust credential plane is designed but not yet implemented. Interfaces may change. Licensed under the [Apache License 2.0](LICENSE). - diff --git a/SECURITY.md b/SECURITY.md index f17109f..3566885 100644 --- a/SECURITY.md +++ b/SECURITY.md @@ -36,7 +36,7 @@ disclosure. We aim to publish fixes and advisories within 90 days of the initial ## Supported Versions | Version | Supported | -|---------|--------------------| +| ------- | ------------------ | | main | :white_check_mark: | Only the latest release and the `main` branch receive security updates. diff --git a/deploy/knative/EXPERIMENTS.md b/deploy/knative/EXPERIMENTS.md index 1bcc745..5e7e184 100644 --- a/deploy/knative/EXPERIMENTS.md +++ b/deploy/knative/EXPERIMENTS.md @@ -4,6 +4,7 @@ Cluster: Kind `sh-knative`, ksvc `serverless-harness` (ns `default`), model `cla Spec: docs/specs/2026-06-25-m7-cluster-experiments-design.md ## E1 — scale-to-zero economics + `E1_RESULT persistent=380 serverless=95 ratio=0.25 pass=yes` Verdict: PASS (PASS = serverless pod-seconds <= 0.6 x persistent) @@ -19,14 +20,16 @@ are roughly constant per turn (independent of idle length) while persistent grow window, so a longer idle yields a larger saving (a short idle can fail the gate). ## E3 — session mobility + A fresh instance recalled the planted token from the Redis log after scale-to-zero. Verdict: PASS ## E4 — crash recovery + After a mid-session pod force-kill, the next turn recalled all completed turns. Verdict: PASS -*Assisted-By: Claude (Anthropic AI) * +_Assisted-By: Claude (Anthropic AI) _ # P3 — Sandbox Sharing-Ratio Results @@ -57,7 +60,7 @@ sandbox work is a **fixed per-leaf constant** — ~2 execs (`.sh-fetch.lock` acq add) at ~280 ms — **independent of review scope**: the shared `/workspace/repo` clone is amortized across leaves, and the small→large review-scope difference lands in the **LLM turn (wall-time / tokens), not in sandbox execs**. So N is roughly **flat** (Kind ~20–24:1, OCP ~12–17:1), not a decreasing curve — -the flat, scope-invariant *shape* holds on both clusters; the absolute ratio scales with per-leaf +the flat, scope-invariant _shape_ holds on both clusters; the absolute ratio scales with per-leaf git-plumbing latency (OCP's EBS-backed `/workspace` is costlier than Kind's → higher duty → lower N). This **supersedes** the earlier single-N figure (N ≈ 29–48:1), which used a trivial `marker.txt` leaf with no real converge. The honest characterization: the harness→sandbox ratio is governed by fixed per-leaf git plumbing — @@ -68,29 +71,29 @@ harness tier, not sandbox occupancy. Non-authoritative (laptop-bound Kind; the authoritative OCP 4.20 run is recorded below). N-vs-workload curve, C=1, warm, 3 samples/variant: -| variant | file | execCount | execMs | wallMs | N (=1/duty) | -|---|---|---|---|---|---| -| L0 | small.py | 2 | 280 | 6095 | 21.8 | -| L1 | medium.py | 2 | 277 | 6648 | 24.0 | -| L2 | large.py | 2 | 290 | 5711 | 19.7 | +| variant | file | execCount | execMs | wallMs | N (=1/duty) | +| ------- | --------- | --------- | ------ | ------ | ----------- | +| L0 | small.py | 2 | 280 | 6095 | 21.8 | +| L1 | medium.py | 2 | 277 | 6648 | 24.0 | +| L2 | large.py | 2 | 290 | 5711 | 19.7 | - **execCount is constant (2) across L0/L1/L2** → per-leaf sandbox duty is fixed git plumbing, not review scope; N ≈ **20–24:1**, flat (the 21.8 / 24 / 19.7 spread is wall-time noise, not a scope trend). -- Concurrency sweep (L2, `max-scale=20`, degradeX=2, 3 samples/rung): c=1→16 throughput 0.114→0.664 leaves/s (monotonically rising), p95 8671→23892 ms. **knee (CAP floor) = 2**, floorPass=false — p95 crossed the 2× baseline bound at c=4 on Kind's single node (latency-bound, *not* throughput saturation). Environment-limited; the authoritative concurrency floor is the deferred OCP run. +- Concurrency sweep (L2, `max-scale=20`, degradeX=2, 3 samples/rung): c=1→16 throughput 0.114→0.664 leaves/s (monotonically rising), p95 8671→23892 ms. **knee (CAP floor) = 2**, floorPass=false — p95 crossed the 2× baseline bound at c=4 on Kind's single node (latency-bound, _not_ throughput saturation). Environment-limited; the authoritative concurrency floor is the deferred OCP run. - Two driver bugs were caught and fixed live (invisible to shellcheck / the gated no-op): a pod-Running race (`wait_ksvc_ready` ≠ a Running pod) and single-pod exec-timing sampling under Knative multi-revision routing (fixed by aggregating the exec-timing delta across all Running harness pods). ### P3.1 authoritative OCP result (OCP 4.20.8, 3-pod pool, image ghcr.io/rossoctl/serverless-harness:0.2.1, SH_MODEL=claude-haiku-4-5, 2026-07-04) Issue #64. Standing P3 stack (4-node cluster, Route ingress); gitd re-applied for the `work` ref. N-vs-workload curve, C=1, warm, 3 samples/variant: -| variant | file | execCount | execMs | wallMs | N (=1/duty) | -|---|---|---|---|---|---| -| L0 | small.py | 2 | 455 | 6751 | 14.8 | -| L1 | medium.py | 2 | 518 | 6551 | 12.6 | -| L2 | large.py | 2 | 431 | 7108 | 16.5 | +| variant | file | execCount | execMs | wallMs | N (=1/duty) | +| ------- | --------- | --------- | ------ | ------ | ----------- | +| L0 | small.py | 2 | 455 | 6751 | 14.8 | +| L1 | medium.py | 2 | 518 | 6551 | 12.6 | +| L2 | large.py | 2 | 431 | 7108 | 16.5 | -- **Confirms the Kind finding: the curve is flat, `execCount` constant at 2 across L0/L1/L2** — per-leaf sandbox work is fixed converge git plumbing, independent of review scope. N ≈ **12–17:1** (the 14.8 / 12.6 / 16.5 spread is git-timing noise). Lower than Kind's ~20–24 only because OCP's EBS-backed `/workspace` makes the ~2 git execs costlier (~470 ms vs ~280 ms), i.e. higher duty — the *shape* is identical. +- **Confirms the Kind finding: the curve is flat, `execCount` constant at 2 across L0/L1/L2** — per-leaf sandbox work is fixed converge git plumbing, independent of review scope. N ≈ **12–17:1** (the 14.8 / 12.6 / 16.5 spread is git-timing noise). Lower than Kind's ~20–24 only because OCP's EBS-backed `/workspace` makes the ~2 git execs costlier (~470 ms vs ~280 ms), i.e. higher duty — the _shape_ is identical. - Concurrency sweep (L2, `max-scale=20`, degradeX=2, 3 samples/rung): c=1→16 throughput 0.122 → 0.152 → 0.232 → 0.223 → 0.591 leaves/s, p95 8118 → 13036 → 17149 → 35712 → 26878 ms. **knee (CAP floor) = 2**, floorPass=false — p95 crossed the 2× baseline bound at c=4 and sustained past c=8. -- **The knee is a *harness-tier* limit, not sandbox saturation.** At the wall, the pinned sandbox is only ~6–8 % busy (duty 0.06–0.08); the p95 blowup under concurrency is LLM latency + Knative cold-start (`max-scale=20` bursts new harness pods), not the sandbox. So the authoritative reading is: **one sandbox comfortably absorbs the offered concurrency (≈6–8 % duty even at C_max); the concurrency ceiling for real code-review leaves is set by the model/harness tier, not the sandbox** — which is exactly the dense-harness / shared-sandbox premise. A recommended `KAGENTI_SANDBOX_CAP` is not sandbox-bound here; scale the harness (`max-scale`) and model throughput first. +- **The knee is a _harness-tier_ limit, not sandbox saturation.** At the wall, the pinned sandbox is only ~6–8 % busy (duty 0.06–0.08); the p95 blowup under concurrency is LLM latency + Knative cold-start (`max-scale=20` bursts new harness pods), not the sandbox. So the authoritative reading is: **one sandbox comfortably absorbs the offered concurrency (≈6–8 % duty even at C_max); the concurrency ceiling for real code-review leaves is set by the model/harness tier, not the sandbox** — which is exactly the dense-harness / shared-sandbox premise. A recommended `KAGENTI_SANDBOX_CAP` is not sandbox-bound here; scale the harness (`max-scale`) and model throughput first. **Bottom line (Kind + OCP):** N is flat and scope-invariant at ~12–24:1 (cluster-dependent on git-plumbing cost), governed by fixed per-leaf git plumbing rather than review scope; and one shared sandbox does not saturate under the offered concurrency — the limiter is the harness/LLM tier. @@ -107,12 +110,12 @@ Non-authoritative — laptop-bound Kind, shape/relative only per design §D5. Au **E6 — saturation curve** (`E6_LADDER="1 2 4 8 16"`, one pinned sandbox, degradeX=2): | c (concurrent leaves) | throughput (leaves/s) | p95 latency (ms) | -|---|---|---| -| 1 | 0.072 | 13881 | -| 2 | 0.158 | 12561 | -| 4 | 0.261 | 15246 | -| 8 | 0.346 | 23020 | -| 16 | 0.608 | 26142 | +| --------------------- | --------------------- | ---------------- | +| 1 | 0.072 | 13881 | +| 2 | 0.158 | 12561 | +| 4 | 0.261 | 15246 | +| 8 | 0.346 | 23020 | +| 16 | 0.608 | 26142 | - **Knee (recommended `KAGENTI_SANDBOX_CAP`): ≥16** — throughput still rising and p95 within the 2× bound at c=16, so no saturation knee was reached below C_max. Treat ≥16 as a floor, not the ceiling. - **Duty cycle (C=1): 0.021** (sandbox-busy execMs≈289 over a ~13.9 s leaf wall) → **derived N ≈ 48:1**. The sandbox is busy ~2 % of leaf wall-clock; the remainder is model/network time in the harness tier. @@ -125,15 +128,17 @@ Non-authoritative — laptop-bound Kind, shape/relative only per design §D5. Au **Takeaway:** the sandbox is lightly used per leaf (~2–6 % duty depending on cold-start), so the harness→sandbox sharing ratio is high (≈20–48:1 on Kind) and one sandbox did not saturate at 16 concurrent leaves — strong support for the dense-harness / shared-sandbox premise. Authoritative CAP/ratio to follow from OCP. ### E6 run host + - ladder: 1 2 4 8 16 - points: [{"c":1,"throughput":0.071,"p95Ms":14050},{"c":2,"throughput":0.245,"p95Ms":8059},{"c":4,"throughput":0.213,"p95Ms":18673},{"c":8,"throughput":0.550,"p95Ms":14424},{"c":16,"throughput":0.594,"p95Ms":26843}] - knee (recommended CAP): 2 -- duty cycle (C=1): 0.035 => derived N ~= 28.6 : 1 +- duty cycle (C=1): 0.035 => derived N ~= 28.6 : 1 - sanity floor (>= 4): false - feed-back max leases/pod at CAP=2: 1 - verdict: no ### E7 run + - refs (distinct, concurrent): 6 - mixed-ref consistency: ok - wall ms: 17387 ; total sandbox exec ms (converge+tools): 0 @@ -146,12 +151,12 @@ Authoritative tier — representative CPU/mem, real EBS RWO, API-server exec, Ro **E6 — saturation curve** (`E6_LADDER="1 2 4 8 16"`, one pinned sandbox, degradeX=2): | c (concurrent leaves) | throughput (leaves/s) | p95 latency (ms) | -|---|---|---| -| 1 | 0.071 | 14050 | -| 2 | 0.245 | 8059 | -| 4 | 0.213 | 18673 | -| 8 | 0.550 | 14424 | -| 16 | 0.594 | 26843 | +| --------------------- | --------------------- | ---------------- | +| 1 | 0.071 | 14050 | +| 2 | 0.245 | 8059 | +| 4 | 0.213 | 18673 | +| 8 | 0.550 | 14424 | +| 16 | 0.594 | 26843 | - Aggregate throughput at c=16 is ~8× c=1 with p95 inside the 2× bound → **the sandbox does not saturate across the tested range**. Duty cycle (C=1) **0.035** (sandbox-busy execMs≈492 over a ~14 s leaf wall) → **derived N ≈ 29:1**. - **Knee/floor caveat:** the detector requires throughput to rise strictly vs the previous rung; per-leaf model-latency + cold-start variance made c=4 (0.213) dip below c=2 (0.245), tripping the break early → `knee=2`, floor=fail on this single run. This is a **measurement-noise artifact, not a capacity ceiling** (throughput climbs ~8× overall). A noise-robust knee (multi-sample rungs and/or a warm `min-scale=1` baseline) is a follow-up. diff --git a/deploy/knative/README-authbridge.md b/deploy/knative/README-authbridge.md index 72a9ff4..525ace4 100644 --- a/deploy/knative/README-authbridge.md +++ b/deploy/knative/README-authbridge.md @@ -4,7 +4,7 @@ This demo shows **Rosso Cortex / AuthBridge** acting as the zero-trust credentia plane on the serverless harness: it does both **credential injection** and **action control** on the harness's two HTTP egress hops, and the real credential is **never held by any model-influenced workload** — only a placeholder is, and the -real value is swapped in at the proxy, *after* an allow/deny gate. +real value is swapped in at the proxy, _after_ an allow/deny gate. The path is gated behind the **`SH_AUTHBRIDGE`** feature flag (off by default). It runs the same way on Kind and OpenShift; only the enable/run entrypoints differ. @@ -88,14 +88,14 @@ The gate resolves the Route automatically (`oc get ksvc serverless-harness -n de ## What it installs -| Component | Manifest | Role | -|-----------|----------|------| -| **AB1** LLM gateway | [`authbridge/ab1-deployment.yaml`](authbridge/ab1-deployment.yaml) | Shared reverse-proxy `Deployment`+`Service`; Hop-1 gate + key injection | -| **ibac-stub** | [`ibac-stub.yaml`](ibac-stub.yaml) | Canned allow/deny policy decision (both hops) | -| tightened egress | [`authbridge/harness-egress-ab1.yaml`](authbridge/harness-egress-ab1.yaml) | `NetworkPolicy` — harness may reach only AB1 (overwrites the base policy) | -| **AB2** sidecar + config | [`authbridge/ab2-config.yaml`](authbridge/ab2-config.yaml), [`sandbox-pool-ab2.yaml`](sandbox-pool-ab2.yaml) | Per-sandbox forward proxy on loopback `:8081`; Hop-2 gate + token injection | -| **echo-target** | [`echo-target.yaml`](echo-target.yaml) | External-API stand-in that reflects the `Authorization` it received | -| secrets | (created by the setup script) | `ab1-llm-cred` (real key), `ab2-egress-cred` (real token); `llm-credentials` repointed to AB1 placeholders | +| Component | Manifest | Role | +| ------------------------ | ------------------------------------------------------------------------------------------------------------ | ---------------------------------------------------------------------------------------------------------- | +| **AB1** LLM gateway | [`authbridge/ab1-deployment.yaml`](authbridge/ab1-deployment.yaml) | Shared reverse-proxy `Deployment`+`Service`; Hop-1 gate + key injection | +| **ibac-stub** | [`ibac-stub.yaml`](ibac-stub.yaml) | Canned allow/deny policy decision (both hops) | +| tightened egress | [`authbridge/harness-egress-ab1.yaml`](authbridge/harness-egress-ab1.yaml) | `NetworkPolicy` — harness may reach only AB1 (overwrites the base policy) | +| **AB2** sidecar + config | [`authbridge/ab2-config.yaml`](authbridge/ab2-config.yaml), [`sandbox-pool-ab2.yaml`](sandbox-pool-ab2.yaml) | Per-sandbox forward proxy on loopback `:8081`; Hop-2 gate + token injection | +| **echo-target** | [`echo-target.yaml`](echo-target.yaml) | External-API stand-in that reflects the `Authorization` it received | +| secrets | (created by the setup script) | `ab1-llm-cred` (real key), `ab2-egress-cred` (real token); `llm-credentials` repointed to AB1 placeholders | On OpenShift these are wired via [`overlays/ocp-authbridge`](overlays/ocp-authbridge) (image remaps + SCC/securityContext patches over the shared manifests); the base, @@ -106,14 +106,14 @@ non-AuthBridge state is [`overlays/ocp`](overlays/ocp). `leaf-smoke.sh` (with `SH_AUTHBRIDGE=1`) proves these before the base leaf claims — Hop-2 first, then Hop-1: -| Claim | What it proves | PASS message | -|-------|----------------|--------------| -| **H2-secret-free** | sandbox holds only the AB2 placeholder | `sandbox spec env holds only the AB2 placeholder + proxy vars, no real cred` | -| **H2-inject** | AB2 allow-path injects the real token at the proxy | `echo reflected the real cred (injected at AB2), sandbox never held it` | -| **H2-deny** | a `tools/call`-shaped request is blocked pre-egress | `denied pre-egress (ibac.no_session or ibac.no_intent ...), no injection` | -| **H1-secret-free** | harness holds only the AB1 placeholder | `harness env holds only the AB1 placeholder + base URL, no real key` | -| **H1-allow** | a leaf completes only because AB1 injected the real key | `allow-path leaf completed (real key injected at AB1)` | -| **H1-deny** | a denylisted call is 403'd at AB1 before injection | `deny-before-inject proven (AB1 returned 403 at ibac, pre-static-inject ...)` | +| Claim | What it proves | PASS message | +| ------------------ | ------------------------------------------------------- | ----------------------------------------------------------------------------- | +| **H2-secret-free** | sandbox holds only the AB2 placeholder | `sandbox spec env holds only the AB2 placeholder + proxy vars, no real cred` | +| **H2-inject** | AB2 allow-path injects the real token at the proxy | `echo reflected the real cred (injected at AB2), sandbox never held it` | +| **H2-deny** | a `tools/call`-shaped request is blocked pre-egress | `denied pre-egress (ibac.no_session or ibac.no_intent ...), no injection` | +| **H1-secret-free** | harness holds only the AB1 placeholder | `harness env holds only the AB1 placeholder + base URL, no real key` | +| **H1-allow** | a leaf completes only because AB1 injected the real key | `allow-path leaf completed (real key injected at AB1)` | +| **H1-deny** | a denylisted call is 403'd at AB1 before injection | `deny-before-inject proven (AB1 returned 403 at ibac, pre-static-inject ...)` | ## See it yourself (manual checks) @@ -167,14 +167,14 @@ clusters.) ## Troubleshooting -| Symptom | Cause / fix | -|---------|-------------| +| Symptom | Cause / fix | +| ------------------------------------------------------------------------------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Harness can't resolve `authbridge-ab1` / DNS times out; LLM calls fail with `Connection timeout` | The egress `NetworkPolicy` is enforced on **both** Kind (modern kindnet) and OCP (OVN-K), so its DNS rule must cover both CoreDNS backends: `kube-system:53` (Kind) and `openshift-dns:5353` (OCP — OVN-K enforces post-DNAT, so 5353 not 53). Both entries are present in [`authbridge/harness-egress-ab1.yaml`](authbridge/harness-egress-ab1.yaml) — see #102 (OCP) and #126 (Kind). | -| `leaf-smoke.sh` exits `SKIP` | Set `LEAF_LIVE_SMOKE=1`. | -| Smoke aborts with an `NS` error | The AuthBridge path requires `NS=default` (AB1 manifests are namespace-pinned). | -| H2 checks see the old sandbox image / no sidecar | The sandbox controller does not roll pods on CR change; the setup/gate force-`delete pod`s the pool and waits. If stale, `kubectl -n default delete pod -l sh.kagenti.io/sandbox-pool=default`. | -| `curl` in the sandbox ignores the proxy for `http://` URLs | It honors lowercase `http_proxy` — the pool sets both cases; use lowercase in ad-hoc curls. | -| sandbox-relay `CrashLoopBackOff` (`ERR_MODULE_NOT_FOUND: tsx`) | Unrelated to AuthBridge; the relay must run from its package dir. Fixed in [`relay-deployment.yaml`](relay-deployment.yaml). | +| `leaf-smoke.sh` exits `SKIP` | Set `LEAF_LIVE_SMOKE=1`. | +| Smoke aborts with an `NS` error | The AuthBridge path requires `NS=default` (AB1 manifests are namespace-pinned). | +| H2 checks see the old sandbox image / no sidecar | The sandbox controller does not roll pods on CR change; the setup/gate force-`delete pod`s the pool and waits. If stale, `kubectl -n default delete pod -l sh.kagenti.io/sandbox-pool=default`. | +| `curl` in the sandbox ignores the proxy for `http://` URLs | It honors lowercase `http_proxy` — the pool sets both cases; use lowercase in ad-hoc curls. | +| sandbox-relay `CrashLoopBackOff` (`ERR_MODULE_NOT_FOUND: tsx`) | Unrelated to AuthBridge; the relay must run from its package dir. Fixed in [`relay-deployment.yaml`](relay-deployment.yaml). | ## Cleanup / restore diff --git a/deploy/knative/README-k8s.md b/deploy/knative/README-k8s.md index 22377a4..e8c4f3b 100644 --- a/deploy/knative/README-k8s.md +++ b/deploy/knative/README-k8s.md @@ -9,15 +9,15 @@ injected via flags — no forked per-cluster YAMLs. ## When to use this vs. setup-kind.sh / setup-ocp.sh -| | `setup-kind.sh` | `setup-k8s.sh` (this) | `setup-ocp.sh` | -|---|---|---|---| -| Target | local Kind (single node) | any real/vanilla K8s cluster | OpenShift 4.x | -| Knative install | raw manifests | raw manifests | OLM operator (OpenShift Serverless) | -| KEDA install | raw manifests | raw manifests (`--with-keda`) | OLM operator (`--with-keda`) | -| Images | builds locally + `kind load` | **prebuilt refs** (`--image`); optional in-cluster build via `setup-shipwright-build.sh` | prebuilt refs / built-in `oc new-build` | -| Namespace | `default` | `--namespace` | `--namespace` | -| StorageClass | cluster default | `--storage-class` (default: cluster default) | cluster default | -| Ingress | Kourier port-forward | port-forward or `--ingress nodeport` | auto-created Route | +| | `setup-kind.sh` | `setup-k8s.sh` (this) | `setup-ocp.sh` | +| --------------- | ---------------------------- | ---------------------------------------------------------------------------------------- | --------------------------------------- | +| Target | local Kind (single node) | any real/vanilla K8s cluster | OpenShift 4.x | +| Knative install | raw manifests | raw manifests | OLM operator (OpenShift Serverless) | +| KEDA install | raw manifests | raw manifests (`--with-keda`) | OLM operator (`--with-keda`) | +| Images | builds locally + `kind load` | **prebuilt refs** (`--image`); optional in-cluster build via `setup-shipwright-build.sh` | prebuilt refs / built-in `oc new-build` | +| Namespace | `default` | `--namespace` | `--namespace` | +| StorageClass | cluster default | `--storage-class` (default: cluster default) | cluster default | +| Ingress | Kourier port-forward | port-forward or `--ingress nodeport` | auto-created Route | Use `setup-kind.sh` for local dev, `setup-ocp.sh` on OpenShift (where OLM manages the operators and Routes are automatic), and **`setup-k8s.sh` for any other Kubernetes** — @@ -109,8 +109,8 @@ harness code for testing/experimentation. Whatever you pick, pass the resulting the cluster that can push to your registry (`--strategy`, default `buildah`; use an insecure-registry variant like `buildah-insecure-direct` for a plain-HTTP in-cluster registry — see [Shipwright's sample strategies](https://github.com/shipwright-io/build/tree/main/samples/buildstrategy)). - Note: an in-cluster registry referenced by ClusterIP/`.svc` must be reachable *and - trusted* by the node container runtime (e.g. listed as an insecure mirror in the node's + Note: an in-cluster registry referenced by ClusterIP/`.svc` must be reachable _and + trusted_ by the node container runtime (e.g. listed as an insecure mirror in the node's registry config) for the kubelet to pull the image back — this is a cluster-level setting, not something either script manages. @@ -174,14 +174,14 @@ scope for this generic script. ## What it installs -| Component | How | -|-----------|-----| +| Component | How | +| ------------------------- | ------------------------------------------------------------------------- | | Knative Serving + Kourier | raw manifests + config patches (autoscaler, PVC/securityContext features) | -| KEDA (optional) | raw manifest (`--with-keda`) | -| agent-sandbox controller | `kubectl apply --server-side` (v0.5.0) | -| Redis | in-repo Deployment (`redis.yaml`) | -| Sandbox pool | `sandbox-pool.yaml`, namespaced + storageClass injected | -| Harness | Knative Service (`service.yaml`) + SA/RBAC | +| KEDA (optional) | raw manifest (`--with-keda`) | +| agent-sandbox controller | `kubectl apply --server-side` (v0.5.0) | +| Redis | in-repo Deployment (`redis.yaml`) | +| Sandbox pool | `sandbox-pool.yaml`, namespaced + storageClass injected | +| Harness | Knative Service (`service.yaml`) + SA/RBAC | ## Cleanup diff --git a/deploy/knative/README-kind.md b/deploy/knative/README-kind.md index 2d5f162..1e39771 100644 --- a/deploy/knative/README-kind.md +++ b/deploy/knative/README-kind.md @@ -49,7 +49,7 @@ curl -H 'Host: serverless-harness.default.example.com' \ `POST /turn` also streams the turn live when the client asks for it with `Accept: text/event-stream`. The default (no `Accept`, or any other value) is unchanged — the same -single JSON body. Streaming is a *representation* of `/turn` chosen by content negotiation, not a +single JSON body. Streaming is a _representation_ of `/turn` chosen by content negotiation, not a separate route. ```bash @@ -102,13 +102,13 @@ loads it into the cluster, falling back to a local build only if the pull is una Environment variables: -| Variable | Default | Description | -|----------|---------|-------------| -| `CLUSTER_NAME` | `sh-knative` | Kind cluster name | -| `KNATIVE_VERSION` | `v1.14.0` | Knative Serving version | -| `SH_IMAGE` | `ghcr.io/rossoctl/serverless-harness:latest` | Published harness image pulled by default (same as `--image`) | -| `FORCE_BUILD` | `false` | Force a local build (same as `--build`) | -| `KEDA_VERSION` | `v2.14.0` | KEDA version | +| Variable | Default | Description | +| ----------------- | -------------------------------------------- | ------------------------------------------------------------- | +| `CLUSTER_NAME` | `sh-knative` | Kind cluster name | +| `KNATIVE_VERSION` | `v1.14.0` | Knative Serving version | +| `SH_IMAGE` | `ghcr.io/rossoctl/serverless-harness:latest` | Published harness image pulled by default (same as `--image`) | +| `FORCE_BUILD` | `false` | Force a local build (same as `--build`) | +| `KEDA_VERSION` | `v2.14.0` | KEDA version | ## Choosing the model @@ -119,7 +119,7 @@ To use a different model, edit `service.yaml` before running the setup script: ```yaml - name: SH_MODEL - value: "claude-sonnet-4-6" # or claude-opus-4-6, claude-haiku-4-5, etc. + value: 'claude-sonnet-4-6' # or claude-opus-4-6, claude-haiku-4-5, etc. ``` Or patch the running Knative Service after deployment: @@ -130,11 +130,11 @@ kubectl set env ksvc/serverless-harness SH_MODEL=claude-sonnet-4-6 This triggers an automatic revision rollout. Available model IDs: -| Model | ID | Notes | -|-------|----|-------| -| Haiku 4.5 | `claude-haiku-4-5` | Default — fast, low cost | -| Sonnet 4.6 | `claude-sonnet-4-6` | Balanced | -| Opus 4.6 | `claude-opus-4-6` | Most capable | +| Model | ID | Notes | +| ---------- | ------------------- | ------------------------ | +| Haiku 4.5 | `claude-haiku-4-5` | Default — fast, low cost | +| Sonnet 4.6 | `claude-sonnet-4-6` | Balanced | +| Opus 4.6 | `claude-opus-4-6` | Most capable | When using a gateway (LiteLLM, etc.), the model ID must match what the gateway accepts — consult your gateway's model routing configuration. @@ -151,15 +151,15 @@ protocol with `SH_MODEL_API`: ```yaml - name: SH_MODEL - value: "ibm-granite/granite-4.1-8b" + value: 'ibm-granite/granite-4.1-8b' - name: SH_MODEL_CUSTOM - value: "1" + value: '1' - name: SH_MODEL_API - value: "openai-completions" + value: 'openai-completions' - name: SH_MODEL_BASE_URL - value: "https:///granite-4-1-8b/v1" + value: 'https:///granite-4-1-8b/v1' - name: OPENAI_API_KEY - value: "" # standard Bearer auth (default) + value: '' # standard Bearer auth (default) ``` For custom-header auth (e.g. IBM RITS's `RITS_API_KEY`), keep the secret in a `secretKeyRef` @@ -167,9 +167,9 @@ env and reference it from `SH_MODEL_HEADERS` via `${VAR}` (the default Bearer is ```yaml - name: SH_MODEL_AUTH - value: "custom-header" + value: 'custom-header' - name: SH_MODEL_HEADERS - value: '{"RITS_API_KEY":"${RITS_API_KEY}"}' # RITS_API_KEY from a secretKeyRef env + value: '{"RITS_API_KEY":"${RITS_API_KEY}"}' # RITS_API_KEY from a secretKeyRef env ``` > Tool-calling is a per-endpoint capability: only routes with the vLLM tool-call parser @@ -177,16 +177,16 @@ env and reference it from `SH_MODEL_HEADERS` via `${VAR}` (the default Bearer is ## What it installs -| Component | How | -|-----------|-----| -| Knative Serving + Kourier | Direct YAML apply from upstream releases | -| KEDA | Direct YAML apply (async leaf ScaledJob support) | -| Knative config | Autoscaler tuning (20s stable-window), PVC feature flags, security-context flag | -| Redis | Lightweight in-repo Deployment (`redis:7-alpine`) | -| Sandbox | Pre-baked image (`sandbox.yaml`, `USER 65532`) | -| `leaf-work` PVC | `ReadWriteOnce`, default StorageClass | -| Harness | Knative Service (`service.yaml`) | -| Ingress | Kourier + port-forward from host | +| Component | How | +| ------------------------- | ------------------------------------------------------------------------------- | +| Knative Serving + Kourier | Direct YAML apply from upstream releases | +| KEDA | Direct YAML apply (async leaf ScaledJob support) | +| Knative config | Autoscaler tuning (20s stable-window), PVC feature flags, security-context flag | +| Redis | Lightweight in-repo Deployment (`redis:7-alpine`) | +| Sandbox | Pre-baked image (`sandbox.yaml`, `USER 65532`) | +| `leaf-work` PVC | `ReadWriteOnce`, default StorageClass | +| Harness | Knative Service (`service.yaml`) | +| Ingress | Kourier + port-forward from host | ## Smoke test @@ -204,12 +204,12 @@ injection + allow/deny control on both harness egress hops — see ## Troubleshooting -| Symptom | Cause / fix | -|---------|-------------| -| `ksvc` never Ready, pod `CrashLoopBackOff` | Check `kubectl logs` — likely missing `llm-credentials` secret or broken image. | -| `/turn` returns `"Connection error"` | The harness can't reach its Anthropic endpoint from the cluster (gateway unreachable). `/health` still works. | -| Image not found after `--skip-build` | Load the image manually: `kind load docker-image dev.local/serverless-harness:local --name sh-knative` | -| Scale-to-zero doesn't happen | Verify `config-autoscaler` settings: `kubectl get cm config-autoscaler -n knative-serving -o yaml` | +| Symptom | Cause / fix | +| ------------------------------------------ | ------------------------------------------------------------------------------------------------------------- | +| `ksvc` never Ready, pod `CrashLoopBackOff` | Check `kubectl logs` — likely missing `llm-credentials` secret or broken image. | +| `/turn` returns `"Connection error"` | The harness can't reach its Anthropic endpoint from the cluster (gateway unreachable). `/health` still works. | +| Image not found after `--skip-build` | Load the image manually: `kind load docker-image dev.local/serverless-harness:local --name sh-knative` | +| Scale-to-zero doesn't happen | Verify `config-autoscaler` settings: `kubectl get cm config-autoscaler -n knative-serving -o yaml` | ## Cleanup diff --git a/deploy/knative/README-ocp.md b/deploy/knative/README-ocp.md index 91e7df5..75b7c62 100644 --- a/deploy/knative/README-ocp.md +++ b/deploy/knative/README-ocp.md @@ -61,15 +61,15 @@ creates a real Route per Knative Service (`oc get ksvc serverless-harness -o jso ## What it installs -| Component | How | -|-----------|-----| -| Knative Serving (+ Kourier) | **Red Hat OpenShift Serverless Operator** (OLM Subscription in `openshift-serverless`) + a `KnativeServing` CR in `knative-serving`. Kourier is bundled. | -| Knative config | Autoscaler tuning + the `podspec-persistent-volume-claim`/`-write`/`-securitycontext` feature flags are set in the **`KnativeServing` CR spec** (the operator reverts direct `config-*` ConfigMap patches). | -| Redis | Lightweight in-repo Deployment (`redis:7-alpine`), runs under `restricted-v2`. | -| Sandbox | Pre-baked image ([`sandbox.Dockerfile`](sandbox.Dockerfile), `USER 65532`), pulled from GHCR (`ghcr.io/rossoctl/serverless-harness-sandbox:latest`, republished by `build.yaml` on every push to `main`; override with `--sandbox-image`). | -| Sandbox `/workspace` PVC | `ReadWriteOnce` (Sandbox CR `volumeClaimTemplates`), cluster-default StorageClass. | -| Harness | Knative Service applied via the [`overlays/ocp`](overlays/ocp) kustomize overlay; SA granted the `nonroot-v2` SCC. | -| Ingress | Auto-created OpenShift Route. | +| Component | How | +| --------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | +| Knative Serving (+ Kourier) | **Red Hat OpenShift Serverless Operator** (OLM Subscription in `openshift-serverless`) + a `KnativeServing` CR in `knative-serving`. Kourier is bundled. | +| Knative config | Autoscaler tuning + the `podspec-persistent-volume-claim`/`-write`/`-securitycontext` feature flags are set in the **`KnativeServing` CR spec** (the operator reverts direct `config-*` ConfigMap patches). | +| Redis | Lightweight in-repo Deployment (`redis:7-alpine`), runs under `restricted-v2`. | +| Sandbox | Pre-baked image ([`sandbox.Dockerfile`](sandbox.Dockerfile), `USER 65532`), pulled from GHCR (`ghcr.io/rossoctl/serverless-harness-sandbox:latest`, republished by `build.yaml` on every push to `main`; override with `--sandbox-image`). | +| Sandbox `/workspace` PVC | `ReadWriteOnce` (Sandbox CR `volumeClaimTemplates`), cluster-default StorageClass. | +| Harness | Knative Service applied via the [`overlays/ocp`](overlays/ocp) kustomize overlay; SA granted the `nonroot-v2` SCC. | +| Ingress | Auto-created OpenShift Route. | Manifests are shared with Kind via the `overlays/ocp` overlay — OpenShift tweaks are kustomize patches, not forked YAMLs. @@ -99,7 +99,7 @@ To use a different model, edit `service.yaml` before running the setup script: ```yaml - name: SH_MODEL - value: "claude-sonnet-4-6" # or claude-opus-4-6, claude-haiku-4-5, etc. + value: 'claude-sonnet-4-6' # or claude-opus-4-6, claude-haiku-4-5, etc. ``` Or patch the running Knative Service after deployment: @@ -110,11 +110,11 @@ oc set env ksvc/serverless-harness SH_MODEL=claude-sonnet-4-6 This triggers an automatic revision rollout. Available model IDs: -| Model | ID | Notes | -|-------|----|-------| -| Haiku 4.5 | `claude-haiku-4-5` | Default — fast, low cost | -| Sonnet 4.6 | `claude-sonnet-4-6` | Balanced | -| Opus 4.6 | `claude-opus-4-6` | Most capable | +| Model | ID | Notes | +| ---------- | ------------------- | ------------------------ | +| Haiku 4.5 | `claude-haiku-4-5` | Default — fast, low cost | +| Sonnet 4.6 | `claude-sonnet-4-6` | Balanced | +| Opus 4.6 | `claude-opus-4-6` | Most capable | When using a gateway (LiteLLM, etc.), the model ID must match what the gateway accepts — consult your gateway's model routing configuration. @@ -131,15 +131,15 @@ protocol with `SH_MODEL_API`: ```yaml - name: SH_MODEL - value: "ibm-granite/granite-4.1-8b" + value: 'ibm-granite/granite-4.1-8b' - name: SH_MODEL_CUSTOM - value: "1" + value: '1' - name: SH_MODEL_API - value: "openai-completions" + value: 'openai-completions' - name: SH_MODEL_BASE_URL - value: "https:///granite-4-1-8b/v1" + value: 'https:///granite-4-1-8b/v1' - name: OPENAI_API_KEY - value: "" # standard Bearer auth (default) + value: '' # standard Bearer auth (default) ``` For custom-header auth (e.g. IBM RITS's `RITS_API_KEY`), keep the secret in a `secretKeyRef` @@ -147,9 +147,9 @@ env and reference it from `SH_MODEL_HEADERS` via `${VAR}` (the default Bearer is ```yaml - name: SH_MODEL_AUTH - value: "custom-header" + value: 'custom-header' - name: SH_MODEL_HEADERS - value: '{"RITS_API_KEY":"${RITS_API_KEY}"}' # RITS_API_KEY from a secretKeyRef env + value: '{"RITS_API_KEY":"${RITS_API_KEY}"}' # RITS_API_KEY from a secretKeyRef env ``` > Tool-calling is a per-endpoint capability: only routes with the vLLM tool-call parser @@ -170,7 +170,7 @@ control on both harness egress hops — see [`README-authbridge.md`](README-auth See [`SMOKE.md`](SMOKE.md#smoke-test-on-openshift) for details. Claims that assert on the **LLM `/turn` response** require the harness to reach its configured -Anthropic endpoint *from the cluster*; health, scale-to-zero/-up, Redis session +Anthropic endpoint _from the cluster_; health, scale-to-zero/-up, Redis session recall, and the 404 path do not. ## Connecting a worker @@ -221,7 +221,7 @@ and verifying the async-leaf path itself on OpenShift is a further step. - **SCC.** The published harness image declares no `USER` (defaults to root), so it runs as an explicit non-root UID (65532) and the script grants the harness ServiceAccount the `nonroot-v2` SCC (`oc adm policy add-scc-to-user nonroot-v2 -z - serverless-harness`). The sandbox image sets `USER 65532` itself and needs no grant. +serverless-harness`). The sandbox image sets `USER 65532` itself and needs no grant. ## Image delivery @@ -246,13 +246,13 @@ on every push to `main`, so OpenShift pulls them directly — no in-cluster buil ## Troubleshooting -| Symptom | Cause / fix | -|---------|-------------| -| `ksvc` never Ready, pod `CreateContainerConfigError: container has runAsNonRoot and image will run as root` | The `nonroot-v2` SCC grant didn't apply. Re-run the script, or `oc adm policy add-scc-to-user nonroot-v2 -z serverless-harness -n `. | -| `ksvc` never Ready, pod `CrashLoopBackOff` with `ERR_MODULE_NOT_FOUND` | The harness image is broken/stale. Use a newer `--image` (the fix shipped in the image build; see the repo history). | -| Sandbox `/workspace` PVC stuck `Pending` | No (default) StorageClass. Set one, or ensure a provisioner is installed. | -| `oc apply -k overlays/ocp` fails with a load-restrictor / "not in or below" error | The overlay references shared base YAMLs one level up. Render with `oc kustomize --load-restrictor LoadRestrictionsNone deploy/knative/overlays/ocp \| oc apply -f -` — `setup-ocp.sh` does this for you. | -| `/turn` returns `"Connection error"` | The harness can't reach its configured Anthropic endpoint from the cluster (egress/gateway reachability). `/health` and session creation still work. | +| Symptom | Cause / fix | +| ----------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `ksvc` never Ready, pod `CreateContainerConfigError: container has runAsNonRoot and image will run as root` | The `nonroot-v2` SCC grant didn't apply. Re-run the script, or `oc adm policy add-scc-to-user nonroot-v2 -z serverless-harness -n `. | +| `ksvc` never Ready, pod `CrashLoopBackOff` with `ERR_MODULE_NOT_FOUND` | The harness image is broken/stale. Use a newer `--image` (the fix shipped in the image build; see the repo history). | +| Sandbox `/workspace` PVC stuck `Pending` | No (default) StorageClass. Set one, or ensure a provisioner is installed. | +| `oc apply -k overlays/ocp` fails with a load-restrictor / "not in or below" error | The overlay references shared base YAMLs one level up. Render with `oc kustomize --load-restrictor LoadRestrictionsNone deploy/knative/overlays/ocp \| oc apply -f -` — `setup-ocp.sh` does this for you. | +| `/turn` returns `"Connection error"` | The harness can't reach its configured Anthropic endpoint from the cluster (egress/gateway reachability). `/health` and session creation still work. | ## Cleanup diff --git a/deploy/knative/README-worker.md b/deploy/knative/README-worker.md index 285f25e..c1775c5 100644 --- a/deploy/knative/README-worker.md +++ b/deploy/knative/README-worker.md @@ -53,7 +53,7 @@ does no matching — leasing stays in the harness pool / `select-sandbox`. ## Step 1 — Set the relay token (required) The relay auth is **fail-closed**. It ships with no token set, so until you -provide one it rejects *every* Attach before parking the stream. Set a token on +provide one it rejects _every_ Attach before parking the stream. Set a token on the relay Deployment: ```bash @@ -89,11 +89,11 @@ leaf through `GrpcRelayTransport`. Copy [`worker-example.yaml`](worker-example.yaml), drop in your image, and apply it. The worker reads three environment variables: -| Variable | Value | Notes | -|----------|-------|-------| -| `SANDBOX_ID` | e.g. `sbx-dev-1` | Stable id; the pool record and presence key are keyed on it. | -| `RELAY_ADDR` | `sandbox-relay.default.svc:8443` | In-cluster relay Service. Plaintext h2c — no TLS in-cluster. | -| `SANDBOX_TOKEN` | `dev-token` | Sent as `authorization: Bearer `. **Must** match Step 1. | +| Variable | Value | Notes | +| --------------- | -------------------------------- | --------------------------------------------------------------- | +| `SANDBOX_ID` | e.g. `sbx-dev-1` | Stable id; the pool record and presence key are keyed on it. | +| `RELAY_ADDR` | `sandbox-relay.default.svc:8443` | In-cluster relay Service. Plaintext h2c — no TLS in-cluster. | +| `SANDBOX_TOKEN` | `dev-token` | Sent as `authorization: Bearer `. **Must** match Step 1. | ```bash # edit worker-example.yaml: set image, SANDBOX_ID, SANDBOX_TOKEN @@ -106,7 +106,7 @@ same id. To run several, give each its own `SANDBOX_ID` (and matching ## Step 4 — Verify -**Presence** — the live Attach stream *is* the registration. Once the worker +**Presence** — the live Attach stream _is_ the registration. Once the worker connects, its id appears in the Redis presence hash and disappears when the stream closes: @@ -213,10 +213,10 @@ reached through a Route rather than a `kourier` port-forward, and images must co from a registry the cluster can pull rather than a kind node's image store. Set three overrides: -| Variable | Why | -|----------|-----| -| `KSVC_URL` | The harness Route. `lib.sh` then targets it directly, drops the `Host` header, and adds `curl -k` for the router's cert. | -| `RELAY_IMAGE` | `relay-deployment.yaml` pins `dev.local/serverless-harness:local`, which exists only in kind. Without this the apply **replaces a working relay with an unpullable one** and aborts at the rollout. | +| Variable | Why | +| -------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | +| `KSVC_URL` | The harness Route. `lib.sh` then targets it directly, drops the `Host` header, and adds `curl -k` for the router's cert. | +| `RELAY_IMAGE` | `relay-deployment.yaml` pins `dev.local/serverless-harness:local`, which exists only in kind. Without this the apply **replaces a working relay with an unpullable one** and aborts at the rollout. | | `WORKER_IMAGE` | A pre-published worker image; skips the `kind load` path. Build one with [`build-image.sh`](../../remote-worker/build-image.sh), which packages a `linux/amd64` binary into the OpenShift internal registry. | ```bash @@ -255,6 +255,7 @@ Four things to know before running it: | sed "s#ghcr.io/rossoctl/serverless-harness:latest##g" \ | oc apply -f - ``` + - **The harness env is flipped, then restored.** The script snapshots the ksvc env, points the pool selector at a label matching no pods so only the worker can be leased, then restores the snapshot exactly. Restore is `trap`-driven on `EXIT`, so @@ -289,7 +290,7 @@ expects: ## Laptop demo: worker as a host container (one command) Everything above runs the worker as a **pod**. That demonstrates the plumbing but not the -driver: the headline claim is a sandbox *outside* the cluster, with **zero inbound rules**, +driver: the headline claim is a sandbox _outside_ the cluster, with **zero inbound rules**, executing a leaf's tool calls. One command shows that on a laptop: ```bash @@ -302,7 +303,7 @@ Teardown removes everything the demo creates, but **asks before deleting the kin `--reuse-cluster` exists so the demo can run against a long-lived dev cluster, and a fresh `--teardown` process cannot know which kind it is looking at. Answer `y`, or pass `--yes` to skip the prompt (`DEMO_ARGS=--yes`). With no terminal to ask on, the cluster is -kept. A run that *did* create the cluster says so on exit and points at `--teardown`; a run +kept. A run that _did_ create the cluster says so on exit and points at `--teardown`; a run against a pre-existing cluster does not. ``` @@ -316,7 +317,7 @@ laptop ``` Neither address is inbound to the laptop. The worker publishes no ports — `docker run` with -no `-p` at all — and reaches the relay only by dialing *out* through +no `-p` at all — and reaches the relay only by dialing _out_ through `kubectl port-forward`. The demo proves the container can reach the tunnel before it starts the worker, and adapts the bind (`--add-host`, then `--address 0.0.0.0`) for runtimes where `host.docker.internal` maps to a bridge IP rather than host loopback. @@ -333,10 +334,10 @@ at the top of this file. The demo defends against it twice: leaf grepping `/etc/os-release` flips its verdict with the backend — and both directions are asserted, so an exec that landed on a pod fails one check or the other: - | backend | pattern `Alpine` | pattern `Red Hat` | model's stated reason | - |---|---|---|---| - | in-cluster sandbox pod | `FLAGGED` | `CLEAR` | "…running Alpine Linux" | - | remote host container | `CLEAR` | `FLAGGED` | "…Red Hat Enterprise Linux 9.8" | + | backend | pattern `Alpine` | pattern `Red Hat` | model's stated reason | + | ---------------------- | ---------------- | ----------------- | ------------------------------- | + | in-cluster sandbox pod | `FLAGGED` | `CLEAR` | "…running Alpine Linux" | + | remote host container | `CLEAR` | `FLAGGED` | "…Red Hat Enterprise Linux 9.8" | The discriminator itself is verified before anything relies on it, and the summary prints the model's own stated reason — so you see the OS it actually read, rather than inferring it @@ -367,7 +368,7 @@ dev value, so nothing is left patched for other callers. - `docker` (or `podman`) + `kind` + `kubectl` + `jq`. **No local Go toolchain** — `remote-worker/Dockerfile` builds the binary in a builder stage. The image is built for - the *host*, never `kind load`ed, so its architecture need not match the kind node. + the _host_, never `kind load`ed, so its architecture need not match the kind node. - A model the **cluster** can reach (`ANTHROPIC_API_KEY`, or `ANTHROPIC_AUTH_TOKEN` + `ANTHROPIC_BASE_URL`). The leaf's verdict is a real model call; the demo fails with an explicit "model endpoint unreachable" message rather than timing out mysteriously. @@ -397,10 +398,10 @@ fiddly; start in-cluster and graduate only if you need external reachability. ## Troubleshooting -| Symptom | Cause / fix | -|---------|-------------| -| Worker connects but the Attach is immediately closed | Token unset or mismatched. Set `SH_RELAY_TOKEN` on the relay (Step 1) and give the worker the same value as `SANDBOX_TOKEN`. Auth is fail-closed. | -| No field in `sh:sandbox:records` | The Attach never succeeded (see above), the worker isn't sending `authorization: Bearer ` metadata, or it isn't sending `Hello` with `sandbox_id` as the first frame. | -| Presence is there but the harness never uses the worker | `SH_REMOTE_SANDBOX` / `SH_RELAY_ADDR` not set on the harness ksvc (Step 2). Confirm with `oc set env ksvc/serverless-harness --list -n default`. | -| A second worker for the same id won't connect | Expected — one live Attach per `SANDBOX_ID`. Give each worker a distinct id. | -| `exit_code` comes back `null` | The child was signalled (or the worker sent `exit_code < 0`). Not an error by itself. | +| Symptom | Cause / fix | +| ------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| Worker connects but the Attach is immediately closed | Token unset or mismatched. Set `SH_RELAY_TOKEN` on the relay (Step 1) and give the worker the same value as `SANDBOX_TOKEN`. Auth is fail-closed. | +| No field in `sh:sandbox:records` | The Attach never succeeded (see above), the worker isn't sending `authorization: Bearer ` metadata, or it isn't sending `Hello` with `sandbox_id` as the first frame. | +| Presence is there but the harness never uses the worker | `SH_REMOTE_SANDBOX` / `SH_RELAY_ADDR` not set on the harness ksvc (Step 2). Confirm with `oc set env ksvc/serverless-harness --list -n default`. | +| A second worker for the same id won't connect | Expected — one live Attach per `SANDBOX_ID`. Give each worker a distinct id. | +| `exit_code` comes back `null` | The child was signalled (or the worker sent `exit_code < 0`). Not an error by itself. | diff --git a/deploy/knative/SMOKE.md b/deploy/knative/SMOKE.md index 4add609..b8e1cb1 100644 --- a/deploy/knative/SMOKE.md +++ b/deploy/knative/SMOKE.md @@ -44,8 +44,8 @@ ## Autoscaler Tuning (dev/testing) ```yaml -stable-window: "20s" # default 60s -scale-to-zero-grace-period: "10s" # default 30s +stable-window: '20s' # default 60s +scale-to-zero-grace-period: '10s' # default 30s ``` For production, use defaults or tune based on cold-start latency tolerance. @@ -64,14 +64,14 @@ For production, use defaults or tune based on cold-start latency tolerance. `setup-ocp.sh` is the OpenShift-native sibling of `setup-kind.sh`. It targets **OpenShift 4.20+** and stands up the same stack, but the OpenShift way (issue #41): -| Concern | Kind (`setup-kind.sh`) | OpenShift (`setup-ocp.sh`) | -|---------|------------------------|-----------------------------| -| Knative | raw upstream YAML + Kourier | **Red Hat OpenShift Serverless Operator** (OLM Subscription) + `KnativeServing` CR (Kourier bundled) | -| Config | `kubectl patch configmap config-*` | feature flags + autoscaler tuning in the `KnativeServing` **CR spec** (the operator reverts ConfigMap patches) | -| Ingress | Kourier port-forward + Host header | auto-created **OpenShift Route** (`ksvc` `status.url`) | +| Concern | Kind (`setup-kind.sh`) | OpenShift (`setup-ocp.sh`) | +| ----------- | ------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------- | +| Knative | raw upstream YAML + Kourier | **Red Hat OpenShift Serverless Operator** (OLM Subscription) + `KnativeServing` CR (Kourier bundled) | +| Config | `kubectl patch configmap config-*` | feature flags + autoscaler tuning in the `KnativeServing` **CR spec** (the operator reverts ConfigMap patches) | +| Ingress | Kourier port-forward + Host header | auto-created **OpenShift Route** (`ksvc` `status.url`) | | Harness UID | hard-coded `runAsUser/fsGroup: 65532` | kept at 65532; the SA is granted the `nonroot-v2` SCC so that explicit non-root UID is admitted (the GHCR image declares no `USER`, so a UID must be set) | -| Sandbox | `alpine` + `apk add` as root | **pre-baked** image (`sandbox.Dockerfile`, sets `USER 65532`) built in-cluster, restricted-v2 compatible | -| Image | `docker build` + `kind load` | published GHCR image (`--image`); sandbox built to the internal registry | +| Sandbox | `alpine` + `apk add` as root | **pre-baked** image (`sandbox.Dockerfile`, sets `USER 65532`) built in-cluster, restricted-v2 compatible | +| Image | `docker build` + `kind load` | published GHCR image (`--image`); sandbox built to the internal registry | Manifests are shared with Kind via the `deploy/knative/overlays/ocp` kustomize overlay (OCP tweaks are patches, not forked YAMLs). @@ -93,7 +93,7 @@ KSVC_URL=$(oc get ksvc serverless-harness -n default -o jsonpath='{.status.url}' `lib.sh` then targets the Route directly (no port-forward, no `Host` header, `-k` for the router cert; Kind behavior is unchanged). Claims that assert on the **LLM `/turn` response** require the harness to reach its configured Anthropic endpoint -*from the cluster*; health, scale-to-zero/-up, Redis session recall, and the 404 +_from the cluster_; health, scale-to-zero/-up, Redis session recall, and the 404 path do not. Observed on OpenShift 4.20 (with private-gateway egress unavailable): **4/6 claims diff --git a/deploy/knative/authbridge/ab1-deployment.yaml b/deploy/knative/authbridge/ab1-deployment.yaml index e128596..65ac1f0 100644 --- a/deploy/knative/authbridge/ab1-deployment.yaml +++ b/deploy/knative/authbridge/ab1-deployment.yaml @@ -60,7 +60,7 @@ spec: # fidelity fixes), digest sha256:c809d5edad2ae41d132328a8e9cdcf5e1924008830f0e73782b1d110154e362a. image: ghcr.io/rossoctl/kagenti-extensions/authbridge:main-9c131ee imagePullPolicy: IfNotPresent - args: ["--config", "/etc/authbridge/config.yaml"] + args: ['--config', '/etc/authbridge/config.yaml'] ports: - containerPort: 8080 volumeMounts: @@ -72,10 +72,10 @@ spec: readOnly: true resources: requests: - memory: "64Mi" - cpu: "50m" + memory: '64Mi' + cpu: '50m' limits: - memory: "128Mi" + memory: '128Mi' volumes: - name: config configMap: diff --git a/deploy/knative/echo-target.yaml b/deploy/knative/echo-target.yaml index 9afdec2..24bdf63 100644 --- a/deploy/knative/echo-target.yaml +++ b/deploy/knative/echo-target.yaml @@ -27,11 +27,11 @@ spec: - containerPort: 8080 resources: requests: - memory: "32Mi" - cpu: "25m" + memory: '32Mi' + cpu: '25m' limits: - memory: "64Mi" - cpu: "100m" + memory: '64Mi' + cpu: '100m' readinessProbe: httpGet: path: / diff --git a/deploy/knative/echo-target/echo.js b/deploy/knative/echo-target/echo.js index ce348de..fd8d46c 100644 --- a/deploy/knative/echo-target/echo.js +++ b/deploy/knative/echo-target/echo.js @@ -6,13 +6,20 @@ const http = require('http'); // cannot forge/split log lines (CodeQL js/log-injection). The reflected JSON response below // is already safe via JSON.stringify. const logSafe = (s) => String(s).replace(/[\r\n]/g, ''); -http.createServer((req, res) => { - let b = ''; - req.on('data', (c) => (b += c)); - req.on('end', () => { - const auth = req.headers['authorization'] || null; - console.log('ECHO-RECV', logSafe(req.method), logSafe(req.url), 'auth=' + logSafe(auth || '(none)')); - res.writeHead(200, { 'Content-Type': 'application/json' }); - res.end(JSON.stringify({ method: req.method, path: req.url, authorization: auth })); - }); -}).listen(8080, () => console.log('echo listening :8080')); +http + .createServer((req, res) => { + let b = ''; + req.on('data', (c) => (b += c)); + req.on('end', () => { + const auth = req.headers['authorization'] || null; + console.log( + 'ECHO-RECV', + logSafe(req.method), + logSafe(req.url), + 'auth=' + logSafe(auth || '(none)'), + ); + res.writeHead(200, { 'Content-Type': 'application/json' }); + res.end(JSON.stringify({ method: req.method, path: req.url, authorization: auth })); + }); + }) + .listen(8080, () => console.log('echo listening :8080')); diff --git a/deploy/knative/gitd.yaml b/deploy/knative/gitd.yaml index 2b9ebc6..0f097b8 100644 --- a/deploy/knative/gitd.yaml +++ b/deploy/knative/gitd.yaml @@ -18,7 +18,7 @@ spec: containers: - name: gitd image: alpine/git:2.45.2 - command: ["/bin/sh", "-c"] + command: ['/bin/sh', '-c'] args: - | set -eu @@ -59,12 +59,12 @@ spec: --base-path=/srv/git --listen=0.0.0.0 --port=9418 /srv/git env: - name: GITD_REFS - value: "16" + value: '16' ports: - containerPort: 9418 resources: - requests: { memory: "32Mi", cpu: "25m" } - limits: { memory: "128Mi" } + requests: { memory: '32Mi', cpu: '25m' } + limits: { memory: '128Mi' } # No securityContext override: the OCP overlay's non-root SCC applies namespace-wide; # git-daemon and git init run fine as an arbitrary non-root UID against /srv/git + /tmp. --- diff --git a/deploy/knative/ibac-stub.yaml b/deploy/knative/ibac-stub.yaml index 1a0d684..1e53886 100644 --- a/deploy/knative/ibac-stub.yaml +++ b/deploy/knative/ibac-stub.yaml @@ -30,16 +30,16 @@ spec: # from knative-server's node_modules loads it fine. Works on the current image, on Kind, and after # the Dockerfile fix republishes. workingDir: /app/packages/knative-server - command: ["node", "--import", "tsx", "/app/packages/ibac-stub/src/main.ts"] + command: ['node', '--import', 'tsx', '/app/packages/ibac-stub/src/main.ts'] env: - name: IBAC_STUB_PORT - value: "8080" + value: '8080' - name: IBAC_STUB_DENY_TOOLS - value: "" + value: '' - name: IBAC_STUB_DENY_URLS - value: "" + value: '' - name: IBAC_STUB_DENY_ARG_MARKERS - value: "" + value: '' ports: - containerPort: 8080 livenessProbe: @@ -52,10 +52,10 @@ spec: port: 8080 resources: requests: - memory: "64Mi" - cpu: "50m" + memory: '64Mi' + cpu: '50m' limits: - memory: "128Mi" + memory: '128Mi' --- apiVersion: v1 kind: Service diff --git a/deploy/knative/leaf-cron.yaml b/deploy/knative/leaf-cron.yaml index dc7a638..5d74149 100644 --- a/deploy/knative/leaf-cron.yaml +++ b/deploy/knative/leaf-cron.yaml @@ -27,11 +27,11 @@ metadata: name: leaf-cron namespace: default spec: - schedule: "0 2 * * *" # daily 02:00; operator edits to taste. Fire on demand via `kubectl create job --from`. - concurrencyPolicy: Forbid # don't start a new dispatcher while the previous fire's dispatcher runs + schedule: '0 2 * * *' # daily 02:00; operator edits to taste. Fire on demand via `kubectl create job --from`. + concurrencyPolicy: Forbid # don't start a new dispatcher while the previous fire's dispatcher runs jobTemplate: spec: - backoffLimit: 1 # one retry of the dispatcher; idempotent because the fire id is stable + backoffLimit: 1 # one retry of the dispatcher; idempotent because the fire id is stable template: spec: restartPolicy: Never @@ -48,13 +48,13 @@ spec: image: dev.local/serverless-harness:local imagePullPolicy: IfNotPresent workingDir: /app/packages/knative-server - command: ["node", "--import", "tsx", "src/cron-dispatch.ts"] + command: ['node', '--import', 'tsx', 'src/cron-dispatch.ts'] securityContext: allowPrivilegeEscalation: false - readOnlyRootFilesystem: true # nothing is written to the root fs; tsx/node use /tmp (emptyDir below) - capabilities: { drop: ["ALL"] } + readOnlyRootFilesystem: true # nothing is written to the root fs; tsx/node use /tmp (emptyDir below) + capabilities: { drop: ['ALL'] } env: - - name: JOB_NAME # the fire id — stable across this Job's pod retries, unique per fire + - name: JOB_NAME # the fire id — stable across this Job's pod retries, unique per fire valueFrom: { fieldRef: { fieldPath: "metadata.labels['job-name']" } } - name: CRON_CONFIG value: /config/schedule.json diff --git a/deploy/knative/leaf-orchestrator.yaml b/deploy/knative/leaf-orchestrator.yaml index d503b19..31c0b65 100644 --- a/deploy/knative/leaf-orchestrator.yaml +++ b/deploy/knative/leaf-orchestrator.yaml @@ -15,7 +15,7 @@ spec: # alpine:3.20 is a tag, not a digest — fine for a local Kind smoke. If this orchestrator # is ever promoted beyond Kind, pin to a digest (alpine@sha256:...) for reproducibility. image: alpine:3.20 - command: ["sh", "-c", "apk add --no-cache curl jq >/dev/null 2>&1 && sleep 100000"] + command: ['sh', '-c', 'apk add --no-cache curl jq >/dev/null 2>&1 && sleep 100000'] volumeMounts: - name: work mountPath: /work diff --git a/deploy/knative/leaf-scaledjob.yaml b/deploy/knative/leaf-scaledjob.yaml index 8090be8..8abc212 100644 --- a/deploy/knative/leaf-scaledjob.yaml +++ b/deploy/knative/leaf-scaledjob.yaml @@ -25,11 +25,11 @@ spec: image: dev.local/serverless-harness:local imagePullPolicy: IfNotPresent workingDir: /app/packages/knative-server - command: ["node", "--import", "tsx", "src/leaf-job.ts"] + command: ['node', '--import', 'tsx', 'src/leaf-job.ts'] securityContext: allowPrivilegeEscalation: false readOnlyRootFilesystem: true - capabilities: { drop: ["ALL"] } + capabilities: { drop: ['ALL'] } # Baseline bounds so a runaway leaf (e.g. large LLM context) can't OOM the node; # maxReplicaCount caps pod count, these cap per-pod usage. Tune to observed workloads. resources: @@ -39,16 +39,16 @@ spec: - name: HOME value: /tmp - name: REDIS_URL - value: "redis://redis.default.svc:6379" + value: 'redis://redis.default.svc:6379' - name: SH_MODEL - value: "claude-haiku-4-5" + value: 'claude-haiku-4-5' # Lease from the shared sandbox pool (least-loaded, Redis-backed) instead # of pinning one pod — otherwise all maxReplicaCount workers collide on # sandbox-0. The harness discovers Running pool pods by this label. - name: KAGENTI_SANDBOX_POOL_SELECTOR - value: "sh.kagenti.io/sandbox-pool=default" + value: 'sh.kagenti.io/sandbox-pool=default' - name: KAGENTI_SANDBOX_NAMESPACE - value: "default" + value: 'default' # Soft per-pod lease ADMISSION cap (back-pressure) — NOT a resource limit: # up to CAP leaves may hold a lease on one pool pod at once. With P pods this # sets target concurrency P*CAP; workers beyond that hit @@ -57,17 +57,19 @@ spec: # high (e.g. 1000) so the lease cap — not the pod — bounds concurrency # (e6-saturation.sh does exactly this, on the sync path). - name: KAGENTI_SANDBOX_CAP - value: "3" + value: '3' - name: KAGENTI_SANDBOX_LEASE_TTL_MS - value: "60000" + value: '60000' - name: LEAF_RESULT_TTL_SECONDS - value: "86400" + value: '86400' - name: ANTHROPIC_API_KEY valueFrom: { secretKeyRef: { name: llm-credentials, key: api-key } } - name: ANTHROPIC_BASE_URL - valueFrom: { secretKeyRef: { name: llm-credentials, key: base-url, optional: true } } + valueFrom: + { secretKeyRef: { name: llm-credentials, key: base-url, optional: true } } - name: ANTHROPIC_AUTH_TOKEN - valueFrom: { secretKeyRef: { name: llm-credentials, key: auth-token, optional: true } } + valueFrom: + { secretKeyRef: { name: llm-credentials, key: auth-token, optional: true } } volumeMounts: - name: tmp mountPath: /tmp @@ -98,11 +100,11 @@ spec: address: redis.default.svc:6379 stream: leaf-queue consumerGroup: leaf-workers - lagCount: "1" - activationLagCount: "0" + lagCount: '1' + activationLagCount: '0' - type: redis-streams metadata: address: redis.default.svc:6379 stream: leaf-queue consumerGroup: leaf-workers - pendingEntriesCount: "1" + pendingEntriesCount: '1' diff --git a/deploy/knative/overlays/ocp-authbridge/patch-ab1.yaml b/deploy/knative/overlays/ocp-authbridge/patch-ab1.yaml index 69e2d93..0ad38c9 100644 --- a/deploy/knative/overlays/ocp-authbridge/patch-ab1.yaml +++ b/deploy/knative/overlays/ocp-authbridge/patch-ab1.yaml @@ -19,4 +19,4 @@ spec: securityContext: allowPrivilegeEscalation: false capabilities: - drop: ["ALL"] + drop: ['ALL'] diff --git a/deploy/knative/overlays/ocp-authbridge/patch-echo-target.yaml b/deploy/knative/overlays/ocp-authbridge/patch-echo-target.yaml index 1b5a36e..ef40f1f 100644 --- a/deploy/knative/overlays/ocp-authbridge/patch-echo-target.yaml +++ b/deploy/knative/overlays/ocp-authbridge/patch-echo-target.yaml @@ -23,4 +23,4 @@ spec: securityContext: allowPrivilegeEscalation: false capabilities: - drop: ["ALL"] + drop: ['ALL'] diff --git a/deploy/knative/overlays/ocp-authbridge/patch-ibac-stub.yaml b/deploy/knative/overlays/ocp-authbridge/patch-ibac-stub.yaml index 367d7ba..f2d3d2e 100644 --- a/deploy/knative/overlays/ocp-authbridge/patch-ibac-stub.yaml +++ b/deploy/knative/overlays/ocp-authbridge/patch-ibac-stub.yaml @@ -21,4 +21,4 @@ spec: securityContext: allowPrivilegeEscalation: false capabilities: - drop: ["ALL"] + drop: ['ALL'] diff --git a/deploy/knative/overlays/ocp-authbridge/patch-sandbox-ab2.yaml b/deploy/knative/overlays/ocp-authbridge/patch-sandbox-ab2.yaml index f0e5b4f..21eea9e 100644 --- a/deploy/knative/overlays/ocp-authbridge/patch-sandbox-ab2.yaml +++ b/deploy/knative/overlays/ocp-authbridge/patch-sandbox-ab2.yaml @@ -24,10 +24,10 @@ value: allowPrivilegeEscalation: false capabilities: - drop: ["ALL"] + drop: ['ALL'] - op: add path: /spec/podTemplate/spec/containers/1/securityContext value: allowPrivilegeEscalation: false capabilities: - drop: ["ALL"] + drop: ['ALL'] diff --git a/deploy/knative/overlays/ocp/patch-sandbox.yaml b/deploy/knative/overlays/ocp/patch-sandbox.yaml index 2316048..5aabe77 100644 --- a/deploy/knative/overlays/ocp/patch-sandbox.yaml +++ b/deploy/knative/overlays/ocp/patch-sandbox.yaml @@ -14,7 +14,7 @@ value: ghcr.io/rossoctl/serverless-harness-sandbox:latest - op: replace path: /spec/podTemplate/spec/containers/0/command - value: ["sleep", "infinity"] + value: ['sleep', 'infinity'] - op: add path: /spec/podTemplate/spec/serviceAccountName value: serverless-harness-sandbox @@ -31,4 +31,4 @@ value: allowPrivilegeEscalation: false capabilities: - drop: ["ALL"] + drop: ['ALL'] diff --git a/deploy/knative/redis.yaml b/deploy/knative/redis.yaml index a037ea0..00b4f61 100644 --- a/deploy/knative/redis.yaml +++ b/deploy/knative/redis.yaml @@ -20,10 +20,10 @@ spec: - containerPort: 6379 resources: requests: - memory: "64Mi" - cpu: "50m" + memory: '64Mi' + cpu: '50m' limits: - memory: "128Mi" + memory: '128Mi' --- apiVersion: v1 kind: Service diff --git a/deploy/knative/relay-deployment.yaml b/deploy/knative/relay-deployment.yaml index aa1bc51..9d2f6ea 100644 --- a/deploy/knative/relay-deployment.yaml +++ b/deploy/knative/relay-deployment.yaml @@ -29,12 +29,12 @@ spec: # From the package dir, tsx and all runtime deps (@grpc/grpc-js, @sh/*) resolve # locally. Matches the package.json `start` script (`node --import tsx src/main.ts`). workingDir: /app/packages/sandbox-relay - command: ["node", "--import", "tsx", "src/main.ts"] + command: ['node', '--import', 'tsx', 'src/main.ts'] env: - name: REDIS_URL value: redis://redis.default.svc.cluster.local:6379 - name: SH_RELAY_PORT - value: "8443" + value: '8443' # Relay auth is fail-closed (see main.ts's makeDefaultValidateToken): with # no token set, every worker Attach is rejected before the stream is parked. # MUST equal the worker's SANDBOX_TOKEN (see worker-example.yaml). This @@ -57,10 +57,10 @@ spec: # footprint; the limit leaves headroom for concurrent relayed streams. resources: requests: - memory: "256Mi" - cpu: "50m" + memory: '256Mi' + cpu: '50m' limits: - memory: "512Mi" + memory: '512Mi' --- apiVersion: v1 kind: Service diff --git a/deploy/knative/sandbox-pool-ab2.yaml b/deploy/knative/sandbox-pool-ab2.yaml index ca25491..641e64d 100644 --- a/deploy/knative/sandbox-pool-ab2.yaml +++ b/deploy/knative/sandbox-pool-ab2.yaml @@ -20,14 +20,14 @@ spec: - metadata: name: workspace spec: - accessModes: ["ReadWriteOnce"] + accessModes: ['ReadWriteOnce'] resources: requests: storage: 1Gi podTemplate: metadata: labels: - sh.kagenti.io/sandbox-pool: default # pool discovery label (harness selects on this) + sh.kagenti.io/sandbox-pool: default # pool discovery label (harness selects on this) spec: # Spread pool pods across nodes (see sandbox-pool.yaml). Soft (ScheduleAnyway) so # single-node clusters still schedule; topologyKey=hostname aims for a 1-pod skew. @@ -46,21 +46,21 @@ spec: # install at startup, so there's no need to bypass the AB2 proxy env below (that # was only ever required for the old apk-at-startup CDN fetch, which raced/timed # out under load and made the pool slow to become ready). - command: ["/bin/sh", "-c", "mkdir -p /workspace && exec sleep infinity"] + command: ['/bin/sh', '-c', 'mkdir -p /workspace && exec sleep infinity'] workingDir: /workspace env: - name: HTTP_PROXY - value: "http://localhost:8081" + value: 'http://localhost:8081' - name: HTTPS_PROXY - value: "http://localhost:8081" + value: 'http://localhost:8081' # curl (used by the H2 smoke claims and the sandbox's own egress) ignores the # uppercase HTTP_PROXY for http:// URLs — it only honors lowercase http_proxy (and # for https:// it honors HTTPS_PROXY). Set both cases so plain `curl http://...` # actually transits AB2 instead of bypassing it. - name: http_proxy - value: "http://localhost:8081" + value: 'http://localhost:8081' - name: https_proxy - value: "http://localhost:8081" + value: 'http://localhost:8081' # NO_PROXY empty on purpose: ALL sandbox HTTP egress transits AB2. Only echo-target # egress happens in this PoC. NOTE: any future DIRECT in-cluster egress from the # sandbox (e.g. redis, the apiserver) would also be routed through AB2, whose @@ -68,24 +68,24 @@ spec: # such hosts must be added to NO_PROXY (this is why the H1-deny probe in leaf-smoke.sh # clears the proxy vars to reach AB1 directly). - name: NO_PROXY - value: "" + value: '' - name: ECHO_CRED - value: "PLACEHOLDER-TOKEN" + value: 'PLACEHOLDER-TOKEN' volumeMounts: - name: workspace mountPath: /workspace resources: requests: - memory: "64Mi" - cpu: "50m" + memory: '64Mi' + cpu: '50m' limits: - memory: "256Mi" + memory: '256Mi' - name: authbridge-ab2 # Official kext image built from kext main (#655 static-inject + #657 reverse-proxy # fidelity fixes), digest sha256:c809d5edad2ae41d132328a8e9cdcf5e1924008830f0e73782b1d110154e362a. image: ghcr.io/rossoctl/kagenti-extensions/authbridge:main-9c131ee imagePullPolicy: IfNotPresent - args: ["--config", "/etc/authbridge/config.yaml"] + args: ['--config', '/etc/authbridge/config.yaml'] volumeMounts: - name: config mountPath: /etc/authbridge/config.yaml @@ -95,10 +95,10 @@ spec: readOnly: true resources: requests: - memory: "64Mi" - cpu: "50m" + memory: '64Mi' + cpu: '50m' limits: - memory: "128Mi" + memory: '128Mi' volumes: - name: config configMap: @@ -120,14 +120,14 @@ spec: - metadata: name: workspace spec: - accessModes: ["ReadWriteOnce"] + accessModes: ['ReadWriteOnce'] resources: requests: storage: 1Gi podTemplate: metadata: labels: - sh.kagenti.io/sandbox-pool: default # pool discovery label (harness selects on this) + sh.kagenti.io/sandbox-pool: default # pool discovery label (harness selects on this) spec: # Spread pool pods across nodes (see sandbox-pool.yaml). Soft (ScheduleAnyway) so # single-node clusters still schedule; topologyKey=hostname aims for a 1-pod skew. @@ -146,21 +146,21 @@ spec: # install at startup, so there's no need to bypass the AB2 proxy env below (that # was only ever required for the old apk-at-startup CDN fetch, which raced/timed # out under load and made the pool slow to become ready). - command: ["/bin/sh", "-c", "mkdir -p /workspace && exec sleep infinity"] + command: ['/bin/sh', '-c', 'mkdir -p /workspace && exec sleep infinity'] workingDir: /workspace env: - name: HTTP_PROXY - value: "http://localhost:8081" + value: 'http://localhost:8081' - name: HTTPS_PROXY - value: "http://localhost:8081" + value: 'http://localhost:8081' # curl (used by the H2 smoke claims and the sandbox's own egress) ignores the # uppercase HTTP_PROXY for http:// URLs — it only honors lowercase http_proxy (and # for https:// it honors HTTPS_PROXY). Set both cases so plain `curl http://...` # actually transits AB2 instead of bypassing it. - name: http_proxy - value: "http://localhost:8081" + value: 'http://localhost:8081' - name: https_proxy - value: "http://localhost:8081" + value: 'http://localhost:8081' # NO_PROXY empty on purpose: ALL sandbox HTTP egress transits AB2. Only echo-target # egress happens in this PoC. NOTE: any future DIRECT in-cluster egress from the # sandbox (e.g. redis, the apiserver) would also be routed through AB2, whose @@ -168,22 +168,22 @@ spec: # such hosts must be added to NO_PROXY (this is why the H1-deny probe in leaf-smoke.sh # clears the proxy vars to reach AB1 directly). - name: NO_PROXY - value: "" + value: '' - name: ECHO_CRED - value: "PLACEHOLDER-TOKEN" + value: 'PLACEHOLDER-TOKEN' volumeMounts: - name: workspace mountPath: /workspace resources: requests: - memory: "64Mi" - cpu: "50m" + memory: '64Mi' + cpu: '50m' limits: - memory: "256Mi" + memory: '256Mi' - name: authbridge-ab2 image: ghcr.io/rossoctl/kagenti-extensions/authbridge:main-9c131ee imagePullPolicy: IfNotPresent - args: ["--config", "/etc/authbridge/config.yaml"] + args: ['--config', '/etc/authbridge/config.yaml'] volumeMounts: - name: config mountPath: /etc/authbridge/config.yaml @@ -193,10 +193,10 @@ spec: readOnly: true resources: requests: - memory: "64Mi" - cpu: "50m" + memory: '64Mi' + cpu: '50m' limits: - memory: "128Mi" + memory: '128Mi' volumes: - name: config configMap: @@ -218,14 +218,14 @@ spec: - metadata: name: workspace spec: - accessModes: ["ReadWriteOnce"] + accessModes: ['ReadWriteOnce'] resources: requests: storage: 1Gi podTemplate: metadata: labels: - sh.kagenti.io/sandbox-pool: default # pool discovery label (harness selects on this) + sh.kagenti.io/sandbox-pool: default # pool discovery label (harness selects on this) spec: # Spread pool pods across nodes (see sandbox-pool.yaml). Soft (ScheduleAnyway) so # single-node clusters still schedule; topologyKey=hostname aims for a 1-pod skew. @@ -244,21 +244,21 @@ spec: # install at startup, so there's no need to bypass the AB2 proxy env below (that # was only ever required for the old apk-at-startup CDN fetch, which raced/timed # out under load and made the pool slow to become ready). - command: ["/bin/sh", "-c", "mkdir -p /workspace && exec sleep infinity"] + command: ['/bin/sh', '-c', 'mkdir -p /workspace && exec sleep infinity'] workingDir: /workspace env: - name: HTTP_PROXY - value: "http://localhost:8081" + value: 'http://localhost:8081' - name: HTTPS_PROXY - value: "http://localhost:8081" + value: 'http://localhost:8081' # curl (used by the H2 smoke claims and the sandbox's own egress) ignores the # uppercase HTTP_PROXY for http:// URLs — it only honors lowercase http_proxy (and # for https:// it honors HTTPS_PROXY). Set both cases so plain `curl http://...` # actually transits AB2 instead of bypassing it. - name: http_proxy - value: "http://localhost:8081" + value: 'http://localhost:8081' - name: https_proxy - value: "http://localhost:8081" + value: 'http://localhost:8081' # NO_PROXY empty on purpose: ALL sandbox HTTP egress transits AB2. Only echo-target # egress happens in this PoC. NOTE: any future DIRECT in-cluster egress from the # sandbox (e.g. redis, the apiserver) would also be routed through AB2, whose @@ -266,22 +266,22 @@ spec: # such hosts must be added to NO_PROXY (this is why the H1-deny probe in leaf-smoke.sh # clears the proxy vars to reach AB1 directly). - name: NO_PROXY - value: "" + value: '' - name: ECHO_CRED - value: "PLACEHOLDER-TOKEN" + value: 'PLACEHOLDER-TOKEN' volumeMounts: - name: workspace mountPath: /workspace resources: requests: - memory: "64Mi" - cpu: "50m" + memory: '64Mi' + cpu: '50m' limits: - memory: "256Mi" + memory: '256Mi' - name: authbridge-ab2 image: ghcr.io/rossoctl/kagenti-extensions/authbridge:main-9c131ee imagePullPolicy: IfNotPresent - args: ["--config", "/etc/authbridge/config.yaml"] + args: ['--config', '/etc/authbridge/config.yaml'] volumeMounts: - name: config mountPath: /etc/authbridge/config.yaml @@ -291,10 +291,10 @@ spec: readOnly: true resources: requests: - memory: "64Mi" - cpu: "50m" + memory: '64Mi' + cpu: '50m' limits: - memory: "128Mi" + memory: '128Mi' volumes: - name: config configMap: diff --git a/deploy/knative/sandbox-pool.yaml b/deploy/knative/sandbox-pool.yaml index b43dbf6..a348320 100644 --- a/deploy/knative/sandbox-pool.yaml +++ b/deploy/knative/sandbox-pool.yaml @@ -15,14 +15,14 @@ spec: - metadata: name: workspace spec: - accessModes: ["ReadWriteOnce"] + accessModes: ['ReadWriteOnce'] resources: requests: storage: 1Gi podTemplate: metadata: labels: - sh.kagenti.io/sandbox-pool: default # pool discovery label (harness selects on this) + sh.kagenti.io/sandbox-pool: default # pool discovery label (harness selects on this) spec: # Spread pool pods across nodes. Pool pods request little CPU/memory, so without a # hint the scheduler bin-packs all N onto one node — capping the pool at a single @@ -39,17 +39,22 @@ spec: containers: - name: sandbox image: alpine:3.20 - command: ["/bin/sh", "-c", "apk add --no-cache bash coreutils findutils grep ripgrep git && mkdir -p /workspace && exec sleep infinity"] + command: + [ + '/bin/sh', + '-c', + 'apk add --no-cache bash coreutils findutils grep ripgrep git && mkdir -p /workspace && exec sleep infinity', + ] workingDir: /workspace volumeMounts: - name: workspace mountPath: /workspace resources: requests: - memory: "64Mi" - cpu: "50m" + memory: '64Mi' + cpu: '50m' limits: - memory: "256Mi" + memory: '256Mi' --- apiVersion: agents.x-k8s.io/v1beta1 @@ -64,14 +69,14 @@ spec: - metadata: name: workspace spec: - accessModes: ["ReadWriteOnce"] + accessModes: ['ReadWriteOnce'] resources: requests: storage: 1Gi podTemplate: metadata: labels: - sh.kagenti.io/sandbox-pool: default # pool discovery label (harness selects on this) + sh.kagenti.io/sandbox-pool: default # pool discovery label (harness selects on this) spec: # Spread pool pods across nodes. Pool pods request little CPU/memory, so without a # hint the scheduler bin-packs all N onto one node — capping the pool at a single @@ -88,17 +93,22 @@ spec: containers: - name: sandbox image: alpine:3.20 - command: ["/bin/sh", "-c", "apk add --no-cache bash coreutils findutils grep ripgrep git && mkdir -p /workspace && exec sleep infinity"] + command: + [ + '/bin/sh', + '-c', + 'apk add --no-cache bash coreutils findutils grep ripgrep git && mkdir -p /workspace && exec sleep infinity', + ] workingDir: /workspace volumeMounts: - name: workspace mountPath: /workspace resources: requests: - memory: "64Mi" - cpu: "50m" + memory: '64Mi' + cpu: '50m' limits: - memory: "256Mi" + memory: '256Mi' --- apiVersion: agents.x-k8s.io/v1beta1 @@ -113,14 +123,14 @@ spec: - metadata: name: workspace spec: - accessModes: ["ReadWriteOnce"] + accessModes: ['ReadWriteOnce'] resources: requests: storage: 1Gi podTemplate: metadata: labels: - sh.kagenti.io/sandbox-pool: default # pool discovery label (harness selects on this) + sh.kagenti.io/sandbox-pool: default # pool discovery label (harness selects on this) spec: # Spread pool pods across nodes. Pool pods request little CPU/memory, so without a # hint the scheduler bin-packs all N onto one node — capping the pool at a single @@ -137,15 +147,19 @@ spec: containers: - name: sandbox image: alpine:3.20 - command: ["/bin/sh", "-c", "apk add --no-cache bash coreutils findutils grep ripgrep git && mkdir -p /workspace && exec sleep infinity"] + command: + [ + '/bin/sh', + '-c', + 'apk add --no-cache bash coreutils findutils grep ripgrep git && mkdir -p /workspace && exec sleep infinity', + ] workingDir: /workspace volumeMounts: - name: workspace mountPath: /workspace resources: requests: - memory: "64Mi" - cpu: "50m" + memory: '64Mi' + cpu: '50m' limits: - memory: "256Mi" - + memory: '256Mi' diff --git a/deploy/knative/sandbox.yaml b/deploy/knative/sandbox.yaml index 9437514..0f2b1d5 100644 --- a/deploy/knative/sandbox.yaml +++ b/deploy/knative/sandbox.yaml @@ -10,7 +10,7 @@ spec: - metadata: name: workspace spec: - accessModes: ["ReadWriteOnce"] + accessModes: ['ReadWriteOnce'] resources: requests: storage: 1Gi @@ -19,14 +19,19 @@ spec: containers: - name: sandbox image: alpine:3.20 - command: ["/bin/sh", "-c", "apk add --no-cache bash coreutils findutils grep ripgrep && mkdir -p /workspace && exec sleep infinity"] + command: + [ + '/bin/sh', + '-c', + 'apk add --no-cache bash coreutils findutils grep ripgrep && mkdir -p /workspace && exec sleep infinity', + ] workingDir: /workspace volumeMounts: - name: workspace mountPath: /workspace resources: requests: - memory: "64Mi" - cpu: "50m" + memory: '64Mi' + cpu: '50m' limits: - memory: "256Mi" + memory: '256Mi' diff --git a/deploy/knative/service.yaml b/deploy/knative/service.yaml index 08f59aa..69652ff 100644 --- a/deploy/knative/service.yaml +++ b/deploy/knative/service.yaml @@ -7,11 +7,11 @@ spec: template: metadata: annotations: - autoscaling.knative.dev/min-scale: "0" - autoscaling.knative.dev/max-scale: "5" - autoscaling.knative.dev/scale-to-zero-pod-retention-period: "30s" - autoscaling.knative.dev/target: "1" - autoscaling.knative.dev/target-burst-capacity: "0" + autoscaling.knative.dev/min-scale: '0' + autoscaling.knative.dev/max-scale: '5' + autoscaling.knative.dev/scale-to-zero-pod-retention-period: '30s' + autoscaling.knative.dev/target: '1' + autoscaling.knative.dev/target-burst-capacity: '0' spec: containerConcurrency: 1 timeoutSeconds: 300 @@ -26,22 +26,22 @@ spec: securityContext: allowPrivilegeEscalation: false readOnlyRootFilesystem: true - capabilities: { drop: ["ALL"] } + capabilities: { drop: ['ALL'] } ports: - containerPort: 8080 env: - name: HOME value: /tmp - name: REDIS_URL - value: "redis://redis.default.svc:6379" + value: 'redis://redis.default.svc:6379' - name: SH_MODEL - value: "claude-haiku-4-5" + value: 'claude-haiku-4-5' - name: KAGENTI_SANDBOX_POOL_SELECTOR - value: "sh.kagenti.io/sandbox-pool=default" + value: 'sh.kagenti.io/sandbox-pool=default' # - name: KAGENTI_SANDBOX_POD # single-pod test override (only effective when KAGENTI_SANDBOX_POOL_SELECTOR is unset; pool selector takes precedence) # value: "sandbox-0-0" - name: LEAF_RESULT_TTL_SECONDS - value: "86400" + value: '86400' - name: ANTHROPIC_API_KEY valueFrom: secretKeyRef: @@ -67,11 +67,11 @@ spec: periodSeconds: 5 resources: requests: - memory: "256Mi" - cpu: "100m" + memory: '256Mi' + cpu: '100m' limits: - memory: "512Mi" - cpu: "500m" + memory: '512Mi' + cpu: '500m' volumeMounts: - name: tmp mountPath: /tmp @@ -91,15 +91,15 @@ metadata: name: serverless-harness-sandbox namespace: default rules: - - apiGroups: [""] - resources: ["pods"] - verbs: ["get", "list"] - - apiGroups: [""] - resources: ["pods/exec"] - verbs: ["create"] - - apiGroups: ["agents.x-k8s.io"] - resources: ["sandboxes"] - verbs: ["get", "list"] + - apiGroups: [''] + resources: ['pods'] + verbs: ['get', 'list'] + - apiGroups: [''] + resources: ['pods/exec'] + verbs: ['create'] + - apiGroups: ['agents.x-k8s.io'] + resources: ['sandboxes'] + verbs: ['get', 'list'] --- apiVersion: rbac.authorization.k8s.io/v1 kind: RoleBinding diff --git a/deploy/knative/swebench-sandbox-pool.yaml b/deploy/knative/swebench-sandbox-pool.yaml index 5fd9cde..fe1494b 100644 --- a/deploy/knative/swebench-sandbox-pool.yaml +++ b/deploy/knative/swebench-sandbox-pool.yaml @@ -27,14 +27,14 @@ spec: - metadata: name: workspace spec: - accessModes: ["ReadWriteOnce"] + accessModes: ['ReadWriteOnce'] resources: requests: storage: 50Gi podTemplate: metadata: labels: - sh.kagenti.io/sandbox-pool: swebench # pool discovery label (harness selects on this) + sh.kagenti.io/sandbox-pool: swebench # pool discovery label (harness selects on this) spec: # Spread pool pods across nodes (see sandbox-pool.yaml). Scoped to the swebench pool # label. Soft (ScheduleAnyway) so single-node clusters still schedule. @@ -56,7 +56,7 @@ spec: - name: sandbox image: image-registry.openshift-image-registry.svc:5000/default/swebench-sandbox:ff962cb83fe5c624-15of15 imagePullPolicy: IfNotPresent - command: ["sleep", "infinity"] + command: ['sleep', 'infinity'] workingDir: /workspace volumeMounts: - name: workspace @@ -65,14 +65,14 @@ spec: # CPU-throttled, matching the repo's emulated-workload pattern; adjust if needed. resources: requests: - memory: "512Mi" - cpu: "250m" + memory: '512Mi' + cpu: '250m' limits: - memory: "4Gi" + memory: '4Gi' securityContext: allowPrivilegeEscalation: false capabilities: - drop: ["ALL"] + drop: ['ALL'] --- apiVersion: agents.x-k8s.io/v1beta1 @@ -87,14 +87,14 @@ spec: - metadata: name: workspace spec: - accessModes: ["ReadWriteOnce"] + accessModes: ['ReadWriteOnce'] resources: requests: storage: 50Gi podTemplate: metadata: labels: - sh.kagenti.io/sandbox-pool: swebench # pool discovery label (harness selects on this) + sh.kagenti.io/sandbox-pool: swebench # pool discovery label (harness selects on this) spec: # Spread pool pods across nodes (see sandbox-pool.yaml). Scoped to the swebench pool # label. Soft (ScheduleAnyway) so single-node clusters still schedule. @@ -116,7 +116,7 @@ spec: - name: sandbox image: image-registry.openshift-image-registry.svc:5000/default/swebench-sandbox:ff962cb83fe5c624-15of15 imagePullPolicy: IfNotPresent - command: ["sleep", "infinity"] + command: ['sleep', 'infinity'] workingDir: /workspace volumeMounts: - name: workspace @@ -125,14 +125,14 @@ spec: # CPU-throttled, matching the repo's emulated-workload pattern; adjust if needed. resources: requests: - memory: "512Mi" - cpu: "250m" + memory: '512Mi' + cpu: '250m' limits: - memory: "4Gi" + memory: '4Gi' securityContext: allowPrivilegeEscalation: false capabilities: - drop: ["ALL"] + drop: ['ALL'] --- apiVersion: agents.x-k8s.io/v1beta1 @@ -147,14 +147,14 @@ spec: - metadata: name: workspace spec: - accessModes: ["ReadWriteOnce"] + accessModes: ['ReadWriteOnce'] resources: requests: storage: 50Gi podTemplate: metadata: labels: - sh.kagenti.io/sandbox-pool: swebench # pool discovery label (harness selects on this) + sh.kagenti.io/sandbox-pool: swebench # pool discovery label (harness selects on this) spec: # Spread pool pods across nodes (see sandbox-pool.yaml). Scoped to the swebench pool # label. Soft (ScheduleAnyway) so single-node clusters still schedule. @@ -176,7 +176,7 @@ spec: - name: sandbox image: image-registry.openshift-image-registry.svc:5000/default/swebench-sandbox:ff962cb83fe5c624-15of15 imagePullPolicy: IfNotPresent - command: ["sleep", "infinity"] + command: ['sleep', 'infinity'] workingDir: /workspace volumeMounts: - name: workspace @@ -185,11 +185,11 @@ spec: # CPU-throttled, matching the repo's emulated-workload pattern; adjust if needed. resources: requests: - memory: "512Mi" - cpu: "250m" + memory: '512Mi' + cpu: '250m' limits: - memory: "4Gi" + memory: '4Gi' securityContext: allowPrivilegeEscalation: false capabilities: - drop: ["ALL"] + drop: ['ALL'] diff --git a/deploy/knative/worker-example.yaml b/deploy/knative/worker-example.yaml index 48e4c70..558ee60 100644 --- a/deploy/knative/worker-example.yaml +++ b/deploy/knative/worker-example.yaml @@ -52,6 +52,6 @@ spec: runAsNonRoot: true allowPrivilegeEscalation: false capabilities: - drop: ["ALL"] + drop: ['ALL'] seccompProfile: type: RuntimeDefault diff --git a/docs/adrs/0000-adr-template.md b/docs/adrs/0000-adr-template.md index 4ce6a41..0c26510 100644 --- a/docs/adrs/0000-adr-template.md +++ b/docs/adrs/0000-adr-template.md @@ -8,8 +8,8 @@ ## Context What forces a decision now? The problem, the constraints, the pressures. State the facts and -the forces at play — not the answer. Enough that a reader a year from now understands *why this -was even a question*. +the forces at play — not the answer. Enough that a reader a year from now understands _why this +was even a question_. ## Decision @@ -34,4 +34,4 @@ knowingly accepted — the honest ones, not just the upside. --- -*Assisted-By: Claude (Anthropic AI) * +_Assisted-By: Claude (Anthropic AI) _ diff --git a/docs/adrs/0001-redis-session-backend.md b/docs/adrs/0001-redis-session-backend.md index 7588703..860fd4a 100644 --- a/docs/adrs/0001-redis-session-backend.md +++ b/docs/adrs/0001-redis-session-backend.md @@ -22,9 +22,9 @@ We will introduce an upstreamable `SessionStorageBackend` seam in Pi core (defau ## Consequences - Positive: A completed turn is durable in Redis; a fresh process resumes by `session_id`; the seam mirrors Pi issue #2032 and stays upstreamable. -- Negative / accepted cost: Fire-and-forget writes can lose an *in-flight* turn on hard kill; durability boundary is the completed turn (flush at `turn_end`/`session_shutdown`). +- Negative / accepted cost: Fire-and-forget writes can lose an _in-flight_ turn on hard kill; durability boundary is the completed turn (flush at `turn_end`/`session_shutdown`). - Follow-up owed: Checkpoint write / compaction path deferred to M4 (M1 implements only the read side). --- -*Assisted-By: Claude (Anthropic AI) * +_Assisted-By: Claude (Anthropic AI) _ diff --git a/docs/adrs/0002-k8s-sandbox-client-remote-exec.md b/docs/adrs/0002-k8s-sandbox-client-remote-exec.md index 7bb3500..f0df22d 100644 --- a/docs/adrs/0002-k8s-sandbox-client-remote-exec.md +++ b/docs/adrs/0002-k8s-sandbox-client-remote-exec.md @@ -27,4 +27,4 @@ We will ship a `@sh/k8s-sandbox` package that routes all seven Pi Operations (re --- -*Assisted-By: Claude (Anthropic AI) * +_Assisted-By: Claude (Anthropic AI) _ diff --git a/docs/adrs/0003-persistent-in-pod-channel.md b/docs/adrs/0003-persistent-in-pod-channel.md index 9e245c8..8e9a855 100644 --- a/docs/adrs/0003-persistent-in-pod-channel.md +++ b/docs/adrs/0003-persistent-in-pod-channel.md @@ -27,4 +27,4 @@ We will add a `persistentExecInPod` transport — a single long-lived `kubectl e --- -*Assisted-By: Claude (Anthropic AI) * +_Assisted-By: Claude (Anthropic AI) _ diff --git a/docs/adrs/0004-knative-serverless-wrapper.md b/docs/adrs/0004-knative-serverless-wrapper.md index 16b4ac0..674772e 100644 --- a/docs/adrs/0004-knative-serverless-wrapper.md +++ b/docs/adrs/0004-knative-serverless-wrapper.md @@ -27,4 +27,4 @@ We will extract a reusable `runTurn()` from `cli.ts` and wrap it in a new `@sh/k --- -*Assisted-By: Claude (Anthropic AI) * +_Assisted-By: Claude (Anthropic AI) _ diff --git a/docs/adrs/0005-mcp-code-mode.md b/docs/adrs/0005-mcp-code-mode.md index bd75823..6873828 100644 --- a/docs/adrs/0005-mcp-code-mode.md +++ b/docs/adrs/0005-mcp-code-mode.md @@ -21,11 +21,11 @@ We will have the model author and run scripts in the sandbox that call MCP over ## Consequences -- Supersedes: the parent research doc's MCP *gateway* (M10) — MCP becomes code run in the sandbox, not a harness-forwarded gateway call. +- Supersedes: the parent research doc's MCP _gateway_ (M10) — MCP becomes code run in the sandbox, not a harness-forwarded gateway call. - Positive: Near-zero Pi surface; large token savings (progressive disclosure + filter-in-code); credentials confined to the waypoint. - Negative / accepted cost: Isolation relaxes to "reachable only through the mediating waypoint"; static MCP roster per sandbox image; HTTP/SSE-only. - Follow-up owed: The unattended actor-token delegation plane (M7–M9) is a hard dependency AuthBridge has not yet wired. --- -*Assisted-By: Claude (Anthropic AI) * +_Assisted-By: Claude (Anthropic AI) _ diff --git a/docs/adrs/0006-generalized-credentialed-egress.md b/docs/adrs/0006-generalized-credentialed-egress.md index 76336c7..7dfd8bd 100644 --- a/docs/adrs/0006-generalized-credentialed-egress.md +++ b/docs/adrs/0006-generalized-credentialed-egress.md @@ -27,4 +27,4 @@ We will extend AuthBridge's placeholder-swap to all HTTP egress: the sandbox hol --- -*Assisted-By: Claude (Anthropic AI) * +_Assisted-By: Claude (Anthropic AI) _ diff --git a/docs/adrs/0007-compaction-checkpoint-fast-path.md b/docs/adrs/0007-compaction-checkpoint-fast-path.md index bb3c0fe..02997d9 100644 --- a/docs/adrs/0007-compaction-checkpoint-fast-path.md +++ b/docs/adrs/0007-compaction-checkpoint-fast-path.md @@ -26,4 +26,4 @@ We will treat Pi's native `compaction` entry as the checkpoint and add an additi --- -*Assisted-By: Claude (Anthropic AI) * +_Assisted-By: Claude (Anthropic AI) _ diff --git a/docs/adrs/0008-experiments-harness.md b/docs/adrs/0008-experiments-harness.md index c6dd937..18d7e14 100644 --- a/docs/adrs/0008-experiments-harness.md +++ b/docs/adrs/0008-experiments-harness.md @@ -26,4 +26,4 @@ We will build a new in-process TypeScript `experiments/` vitest workspace where --- -*Assisted-By: Claude (Anthropic AI) * +_Assisted-By: Claude (Anthropic AI) _ diff --git a/docs/adrs/0009-cluster-experiments.md b/docs/adrs/0009-cluster-experiments.md index c23d2df..03337f9 100644 --- a/docs/adrs/0009-cluster-experiments.md +++ b/docs/adrs/0009-cluster-experiments.md @@ -26,4 +26,4 @@ We will implement the three experiments as idempotent bash drivers under `deploy --- -*Assisted-By: Claude (Anthropic AI) * +_Assisted-By: Claude (Anthropic AI) _ diff --git a/docs/adrs/0010-identity-spine.md b/docs/adrs/0010-identity-spine.md index 4d8d9cf..82f7e19 100644 --- a/docs/adrs/0010-identity-spine.md +++ b/docs/adrs/0010-identity-spine.md @@ -27,4 +27,4 @@ We will mint a **per-session** SPIFFE identity with the attested user encoded in --- -*Assisted-By: Claude (Anthropic AI) * +_Assisted-By: Claude (Anthropic AI) _ diff --git a/docs/adrs/0011-harness-lockdown.md b/docs/adrs/0011-harness-lockdown.md index da22113..566986b 100644 --- a/docs/adrs/0011-harness-lockdown.md +++ b/docs/adrs/0011-harness-lockdown.md @@ -26,4 +26,4 @@ We will defend the harness by making any local execution unrewarding and unable --- -*Assisted-By: Claude (Anthropic AI) * +_Assisted-By: Claude (Anthropic AI) _ diff --git a/docs/adrs/0012-inference-injector.md b/docs/adrs/0012-inference-injector.md index 5be9fe6..e67884e 100644 --- a/docs/adrs/0012-inference-injector.md +++ b/docs/adrs/0012-inference-injector.md @@ -27,4 +27,4 @@ We will run a separate, shared, long-lived injector pod that concretely realizes --- -*Assisted-By: Claude (Anthropic AI) * +_Assisted-By: Claude (Anthropic AI) _ diff --git a/docs/adrs/0013-leaf-session-backend-reprioritization.md b/docs/adrs/0013-leaf-session-backend-reprioritization.md index 70ce0ae..02ddbe6 100644 --- a/docs/adrs/0013-leaf-session-backend-reprioritization.md +++ b/docs/adrs/0013-leaf-session-backend-reprioritization.md @@ -27,4 +27,4 @@ We will treat the harness's core role as an excellent leaf-session backend invok --- -*Assisted-By: Claude (Anthropic AI) * +_Assisted-By: Claude (Anthropic AI) _ diff --git a/docs/adrs/0014-mvp-leaf-session-contract.md b/docs/adrs/0014-mvp-leaf-session-contract.md index 0ffb820..f3cad14 100644 --- a/docs/adrs/0014-mvp-leaf-session-contract.md +++ b/docs/adrs/0014-mvp-leaf-session-contract.md @@ -27,4 +27,4 @@ We will build a job-mode `/runs` endpoint that runs an agent autonomously to a s --- -*Assisted-By: Claude (Anthropic AI) * +_Assisted-By: Claude (Anthropic AI) _ diff --git a/docs/adrs/0015-async-leaf-completion.md b/docs/adrs/0015-async-leaf-completion.md index 48e9dc3..a69363d 100644 --- a/docs/adrs/0015-async-leaf-completion.md +++ b/docs/adrs/0015-async-leaf-completion.md @@ -33,4 +33,4 @@ done-marker on the orchestrator's volume. --- -*Assisted-By: Claude (Anthropic AI) * +_Assisted-By: Claude (Anthropic AI) _ diff --git a/docs/adrs/0016-human-gate.md b/docs/adrs/0016-human-gate.md index 71c1a22..42636e1 100644 --- a/docs/adrs/0016-human-gate.md +++ b/docs/adrs/0016-human-gate.md @@ -34,4 +34,4 @@ agent with a continuation prompt derived from the decision. --- -*Assisted-By: Claude (Anthropic AI) * +_Assisted-By: Claude (Anthropic AI) _ diff --git a/docs/adrs/0017-registry-securitycontext-hardening.md b/docs/adrs/0017-registry-securitycontext-hardening.md index 63b52e3..608268c 100644 --- a/docs/adrs/0017-registry-securitycontext-hardening.md +++ b/docs/adrs/0017-registry-securitycontext-hardening.md @@ -33,4 +33,4 @@ existing gate smoke. --- -*Assisted-By: Claude (Anthropic AI) * +_Assisted-By: Claude (Anthropic AI) _ diff --git a/docs/adrs/0018-scheduled-leaf-dispatch.md b/docs/adrs/0018-scheduled-leaf-dispatch.md index e7b3b9b..a3097ad 100644 --- a/docs/adrs/0018-scheduled-leaf-dispatch.md +++ b/docs/adrs/0018-scheduled-leaf-dispatch.md @@ -31,4 +31,4 @@ ConfigMap-defined static envelope list, substitutes the fire id (its owning Job --- -*Assisted-By: Claude (Anthropic AI) * +_Assisted-By: Claude (Anthropic AI) _ diff --git a/docs/adrs/0019-ocp-fs-free-deployment.md b/docs/adrs/0019-ocp-fs-free-deployment.md index 0e45e01..ef96a1a 100644 --- a/docs/adrs/0019-ocp-fs-free-deployment.md +++ b/docs/adrs/0019-ocp-fs-free-deployment.md @@ -32,4 +32,4 @@ agent-sandbox `Sandbox` CR's `volumeClaimTemplates`, and run both harness and sa --- -*Assisted-By: Claude (Anthropic AI) * +_Assisted-By: Claude (Anthropic AI) _ diff --git a/docs/adrs/0020-fs-free-harness.md b/docs/adrs/0020-fs-free-harness.md index 68efda8..0b14b15 100644 --- a/docs/adrs/0020-fs-free-harness.md +++ b/docs/adrs/0020-fs-free-harness.md @@ -26,4 +26,4 @@ We will make the harness perform zero filesystem I/O: move leaf inputs and the r --- -*Assisted-By: Claude (Anthropic AI) * +_Assisted-By: Claude (Anthropic AI) _ diff --git a/docs/adrs/0021-shared-sandbox-pool.md b/docs/adrs/0021-shared-sandbox-pool.md index f9f8695..566ef02 100644 --- a/docs/adrs/0021-shared-sandbox-pool.md +++ b/docs/adrs/0021-shared-sandbox-pool.md @@ -27,4 +27,4 @@ We will run the pool as N distinct single-instance `Sandbox` CRs sharing a commo --- -*Assisted-By: Claude (Anthropic AI) * +_Assisted-By: Claude (Anthropic AI) _ diff --git a/docs/adrs/0022-workload-parameterized-sandbox-load.md b/docs/adrs/0022-workload-parameterized-sandbox-load.md index 455224e..c0aea74 100644 --- a/docs/adrs/0022-workload-parameterized-sandbox-load.md +++ b/docs/adrs/0022-workload-parameterized-sandbox-load.md @@ -26,4 +26,4 @@ We will make the headline output an N-vs-workload curve measured confound-free a --- -*Assisted-By: Claude (Anthropic AI) * +_Assisted-By: Claude (Anthropic AI) _ diff --git a/docs/adrs/0023-sandbox-sharing-ratio-experiments.md b/docs/adrs/0023-sandbox-sharing-ratio-experiments.md index 8bdd574..3b8a8cb 100644 --- a/docs/adrs/0023-sandbox-sharing-ratio-experiments.md +++ b/docs/adrs/0023-sandbox-sharing-ratio-experiments.md @@ -26,4 +26,4 @@ We will measure the sharing ratio and per-sandbox concurrency cap empirically on --- -*Assisted-By: Claude (Anthropic AI) * +_Assisted-By: Claude (Anthropic AI) _ diff --git a/docs/adrs/0024-sandbox-transport-remote-exec.md b/docs/adrs/0024-sandbox-transport-remote-exec.md index 1f7de85..81b1158 100644 --- a/docs/adrs/0024-sandbox-transport-remote-exec.md +++ b/docs/adrs/0024-sandbox-transport-remote-exec.md @@ -7,7 +7,7 @@ ## Context -The harness runs every Pi tool call inside a sandbox pod via `kubectl exec`, so it must dial *into* the pod through the kube API. That rules out any sandbox behind NAT, on-prem, on a laptop, or in another cloud — and blocks the top driver, bring-your-own (untrusted third-party) sandboxes. Reaching those requires inverting connectivity (the sandbox dials *out*) with a contract that is language-neutral (any runtime can host a worker) and firewall-friendly (one outbound TLS connection on `:443`), without touching the Pi loop, the session backend, or the leaf queue. An earlier revision of this PR got the outbound-dial direction right but carried the RPC over Redis Streams behind a TypeScript interface — locking workers to TS via JSON+base64 frames and forcing Redis (a port `:443`-only egress commonly blocks) into the exec path. +The harness runs every Pi tool call inside a sandbox pod via `kubectl exec`, so it must dial _into_ the pod through the kube API. That rules out any sandbox behind NAT, on-prem, on a laptop, or in another cloud — and blocks the top driver, bring-your-own (untrusted third-party) sandboxes. Reaching those requires inverting connectivity (the sandbox dials _out_) with a contract that is language-neutral (any runtime can host a worker) and firewall-friendly (one outbound TLS connection on `:443`), without touching the Pi loop, the session backend, or the leaf queue. An earlier revision of this PR got the outbound-dial direction right but carried the RPC over Redis Streams behind a TypeScript interface — locking workers to TS via JSON+base64 frames and forcing Redis (a port `:443`-only egress commonly blocks) into the exec path. ## Decision @@ -70,7 +70,7 @@ is a third `SandboxTransport`, Read/Write/Edit/Ls/Find, so the file-reading tools are exactly the ones running without a cap. The battery therefore covers two of three implementations, and Pi can still tell the backends apart on output volume. Capping the Read path is a production behaviour change and is tracked -separately. What *is* closed here is the damaging consequence: because that transport falls +separately. What _is_ closed here is the damaging consequence: because that transport falls back to the capped `KubectlTransport` on channel death, a truncated read could reach Pi's Edit tool and be written back over the file, so `createPodReadOps.readFile` now throws instead of returning bytes it cannot vouch for. @@ -128,4 +128,4 @@ than silently skipped, because quietly omitting a case for one implementation is --- -*Assisted-By: Claude (Anthropic AI) * +_Assisted-By: Claude (Anthropic AI) _ diff --git a/docs/adrs/0025-authbridge-deployment-topology.md b/docs/adrs/0025-authbridge-deployment-topology.md index e15d0a6..f702d26 100644 --- a/docs/adrs/0025-authbridge-deployment-topology.md +++ b/docs/adrs/0025-authbridge-deployment-topology.md @@ -10,15 +10,15 @@ The zero-trust credential plane (Phase 2, `Z`-prefix) is being reframed around **Rosso Cortex / AuthBridge** as the concrete injection **and** control mechanism, for a single-tenant PoC. AuthBridge is -a Go HTTP plugin pipeline that can run credential injection (`token-broker`/`token-exchange`) *and* +a Go HTTP plugin pipeline that can run credential injection (`token-broker`/`token-exchange`) _and_ control plugins (`SPARC` grounding, `IBAC` intent) on the same egress hop. Adopting it forces a question the earlier specs did not settle uniformly: **where does AuthBridge physically sit?** Two harness egress hops need it, and they differ: - **harness → LLM** — one fixed destination, one shared single-tenant provider key. Z3 (inference - injector) had specified a **shared reverse-proxy pod** but explicitly *rejected* AuthBridge in favor of - a plain Go injector — a choice made when only *credential injection* was in scope. Once **control + injector) had specified a **shared reverse-proxy pod** but explicitly _rejected_ AuthBridge in favor of + a plain Go injector — a choice made when only _credential injection_ was in scope. Once **control plugins (SPARC/IBAC) are also wanted on this hop**, a plain injector is insufficient and AuthBridge is justified — which reopens the placement question. - **sandbox → external API** — arbitrary destinations; Z5 (generalized credentialed egress) already @@ -27,8 +27,8 @@ Two harness egress hops need it, and they differ: own outbound calls. The forces: the harness is serverless/scale-to-zero (Knative); Kubernetes `NetworkPolicy` selects -*pods*, not containers; AuthBridge's in-memory `abph_` placeholder-swap is single-process only; kagenti's -target mesh is Istio **ambient** (L7 policy at a shared *waypoint*, not per-pod sidecars); and per-caller +_pods_, not containers; AuthBridge's in-memory `abph_` placeholder-swap is single-process only; kagenti's +target mesh is Istio **ambient** (L7 policy at a shared _waypoint_, not per-pod sidecars); and per-caller identity attribution depends on the deferred Z1 identity spine. ## Decision @@ -43,7 +43,7 @@ gateway holds no key and is not bound by the single-process `abph_` limitation. ### Alternatives considered - **AuthBridge #1 as a per-harness sidecar** (`envoy-sidecar`/co-located) — rejected for the shared role: - a `NetworkPolicy` cannot stop the *harness pod* from egressing when the proxy shares its network + a `NetworkPolicy` cannot stop the _harness pod_ from egressing when the proxy shares its network namespace, so the enforceable "harness pod has zero public egress" boundary (Z2 L3) is lost; it also scales a proxy with every scale-to-zero replica and diverges from the ambient/waypoint target shape. (It is simpler — loopback, no harness↔gateway mTLS, local session context — and remains a valid @@ -57,8 +57,8 @@ gateway holds no key and is not bound by the single-process `abph_` limitation. ## Consequences - Positive: the harness pod can be locked to **zero public egress** (default-deny except → the gateway - `Service`), giving an enforceable "no key *and* can't phone home" boundary (Z2); a single shared audit - and policy chokepoint for all harness→LLM traffic; alignment with the kagenti Istio-ambient *waypoint* + `Service`), giving an enforceable "no key _and_ can't phone home" boundary (Z2); a single shared audit + and policy chokepoint for all harness→LLM traffic; alignment with the kagenti Istio-ambient _waypoint_ production shape; the sandbox gate works identically whether the sandbox is in-cluster or BYO; because injection is via the stateless `token-broker`, the shared gateway can scale horizontally and no workload ever holds the real key (it lives only in `static-broker`). @@ -74,4 +74,4 @@ gateway holds no key and is not bound by the single-process `abph_` limitation. --- -*Assisted-By: Claude (Anthropic AI) * +_Assisted-By: Claude (Anthropic AI) _ diff --git a/docs/adrs/0026-rc1-static-inject-plugin.md b/docs/adrs/0026-rc1-static-inject-plugin.md index bef9eaf..0ccc614 100644 --- a/docs/adrs/0026-rc1-static-inject-plugin.md +++ b/docs/adrs/0026-rc1-static-inject-plugin.md @@ -13,7 +13,7 @@ new **`static-broker` HTTP service** (`POST /sessions/token`, keyed by `X-Server token). Two facts surfaced during RC1-0 planning made that a poor fit for a single-tenant, static PoC: 1. The shipped `token-broker` contract is **OAuth-issuance-shaped** — it forwards the caller's inbound JWT - and expects the broker to *issue* a target-service token. That is heavier than a static single-tenant + and expects the broker to _issue_ a target-service token. That is heavier than a static single-tenant swap, and `token-broker` is not even present in the `authbridge-lite` binary. 2. A separate `static-broker` HTTP service adds a moving part whose only job is to hand back a static secret — a network hop and a deployment surface with no PoC value. @@ -61,4 +61,4 @@ is the only place the real credential lives. --- -*Assisted-By: Claude (Anthropic AI) * +_Assisted-By: Claude (Anthropic AI) _ diff --git a/docs/adrs/0027-rc1-control-gate-and-hop2-realization.md b/docs/adrs/0027-rc1-control-gate-and-hop2-realization.md index 132328e..37b66c7 100644 --- a/docs/adrs/0027-rc1-control-gate-and-hop2-realization.md +++ b/docs/adrs/0027-rc1-control-gate-and-hop2-realization.md @@ -58,4 +58,4 @@ For the RC1 PoC: --- -*Assisted-By: Claude (Anthropic AI) * +_Assisted-By: Claude (Anthropic AI) _ diff --git a/docs/adrs/0028-async-prompt-dispatch.md b/docs/adrs/0028-async-prompt-dispatch.md index 787ad23..0839760 100644 --- a/docs/adrs/0028-async-prompt-dispatch.md +++ b/docs/adrs/0028-async-prompt-dispatch.md @@ -37,7 +37,7 @@ default`) and inherits `/turn`'s `resolveSandboxConfig` sandbox routing. ## Consequences -- Positive: a prompt behaves identically sync (`/turn`, `async:false`) or async (`async:true`) because it runs the *same* code; the `/runs` route, job runner, and KEDA `ScaledJob` are unchanged; prompt leaves inherit async resumability for free from the durable session log. +- Positive: a prompt behaves identically sync (`/turn`, `async:false`) or async (`async:true`) because it runs the _same_ code; the `/runs` route, job runner, and KEDA `ScaledJob` are unchanged; prompt leaves inherit async resumability for free from the durable session log. - Negative / accepted cost: prompt leaves get no per-leaf pool isolation (they share `/turn`'s sandbox model), so a fleet of async prompts is not lease-bounded the way solve leaves are; `runTurn` is refactored, so its behavior is now pinned by a regression test rather than by being the only caller. A prompt leaf may still be addressed to a `workloadId` (the workload gates existence and returns 404 if absent), but its pool selector is intentionally ignored — the API boundary logs a warning rather than injecting a selector that `executeTurn` would silently drop. - Follow-up owed: pool-based isolation (a `selectPoolSandbox` lease) for prompt leaves, deferred until a driver needs it; extend `deploy/knative/leaf-async-smoke.sh` with a `responded` claim. @@ -80,4 +80,4 @@ async queue behaves the same as before. --- -*Assisted-By: Claude (Anthropic AI) * +_Assisted-By: Claude (Anthropic AI) _ diff --git a/docs/adrs/0029-turn-sse-streaming.md b/docs/adrs/0029-turn-sse-streaming.md index 1d57e5a..775d94c 100644 --- a/docs/adrs/0029-turn-sse-streaming.md +++ b/docs/adrs/0029-turn-sse-streaming.md @@ -21,7 +21,7 @@ the non-streaming response must stay byte-for-byte identical. We will add streaming as a **representation of `/turn` selected by content negotiation** (`Accept: text/event-stream`), **not** a new `/turn/stream` route. Both modes run the **same** -`executeTurn` core (the shared turn engine ADR-0028 extracted); streaming adds only an event *sink*, +`executeTurn` core (the shared turn engine ADR-0028 extracted); streaming adds only an event _sink_, not a second engine. The sink is a new Pi extension factory `sseExtension(onEvent)` in `harness/src/turn-stream.ts` — the same shape as `flushExtension` — that translates Pi session events into a neutral `TurnStreamFrame` union. `executeTurn` gains two **optional, additive** inputs, @@ -52,4 +52,4 @@ activator) and, crucially, **validated** by a gated smoke that asserts inter-fra --- -*Assisted-By: Claude (Anthropic AI) * +_Assisted-By: Claude (Anthropic AI) _ diff --git a/docs/adrs/README.md b/docs/adrs/README.md index 1416fb8..3064301 100644 --- a/docs/adrs/README.md +++ b/docs/adrs/README.md @@ -9,48 +9,48 @@ we accepted — in a form that stays true even as the code around it changes. Reconstructed from the design specs in [`../specs/`](../specs/) (each ADR links back to its spec). Chronological by the spec's date; numbers are permanent. -| # | Decision | Status | -|---|----------|--------| -| [0001](0001-redis-session-backend.md) | Persist Pi session state through a pluggable Redis backend | Implemented | -| [0002](0002-k8s-sandbox-client-remote-exec.md) | Route Pi tool execution to a remote pod via `kubectl exec` | Implemented | -| [0003](0003-persistent-in-pod-channel.md) | Reuse a long-lived in-pod bash session for fast tool ops | Implemented | -| [0004](0004-knative-serverless-wrapper.md) | Expose the harness as a scale-to-zero Knative HTTP service | Implemented | -| [0005](0005-mcp-code-mode.md) | Originate MCP calls as code the model runs in the sandbox | Accepted | -| [0006](0006-generalized-credentialed-egress.md) | Placeholder-swap over a forward proxy for all credentialed egress | Accepted | -| [0007](0007-compaction-checkpoint-fast-path.md) | Ride Pi's native compaction entry as the resume checkpoint | Implemented | -| [0008](0008-experiments-harness.md) | Measure the loader as in-process reconstruction cost, not end-to-end latency | Implemented | -| [0009](0009-cluster-experiments.md) | Drive cluster experiments with bash extending `smoke.sh` | Implemented | -| [0010](0010-identity-spine.md) | Per-session SPIFFE identity, user in the attested path, minted by the orchestrator | Accepted | -| [0011](0011-harness-lockdown.md) | Defend the harness by defanging local execution, not by mediating egress | Accepted | -| [0012](0012-inference-injector.md) | A separate injector pod holds the provider key and owns public LLM egress | Accepted | -| [0013](0013-leaf-session-backend-reprioritization.md) | Reprioritize the harness as a leaf-session backend; defer the heavy credential plane | Accepted | -| [0014](0014-mvp-leaf-session-contract.md) | Prove the backend with a run-to-completion `/runs` invocation contract | Implemented | -| [0015](0015-async-leaf-completion.md) | KEDA ScaledJob over a Redis Streams queue for background leaf execution | Implemented | -| [0016](0016-human-gate.md) | Human-gate as a structured terminal plus externally-triggered continuation | Implemented | -| [0017](0017-registry-securitycontext-hardening.md) | Apply the non-root least-privilege securityContext baseline to agent pods | Implemented | -| [0018](0018-scheduled-leaf-dispatch.md) | Native Kubernetes CronJob as the scheduled dispatch start signal | Implemented | -| [0019](0019-ocp-fs-free-deployment.md) | Durable RWO EBS Sandbox CR, non-root under nonroot-v2, for the OCP deployment | Implemented | -| [0020](0020-fs-free-harness.md) | Filesystem-free harness (envelope inline + Redis, working set on Sandbox CR) | Implemented | -| [0021](0021-shared-sandbox-pool.md) | Shared sandbox pool via N Sandbox CRs with harness-side Redis-lease routing | Implemented | -| [0022](0022-workload-parameterized-sandbox-load.md) | Report the sharing ratio as an N-vs-workload curve, not a single number | Implemented | -| [0023](0023-sandbox-sharing-ratio-experiments.md) | Measure sandbox sharing capacity on runc; split Kata isolation into P4 | Implemented | -| [0024](0024-sandbox-transport-remote-exec.md) | Remote sandbox exec over a worker-dialed gRPC stream, contract as language-neutral Protobuf | Accepted | -| [0025](0025-authbridge-deployment-topology.md) | AuthBridge topology: shared LLM-egress gateway (AB1) + per-sandbox egress proxy (AB2) | Accepted | -| [0026](0026-rc1-static-inject-plugin.md) | RC1 credential injection via a dedicated `static-inject` plugin (not a broker service) | Accepted | -| [0027](0027-rc1-control-gate-and-hop2-realization.md) | RC1 control gate as IBAC-only; Hop-2 egress interception over plain HTTP | Accepted | -| [0028](0028-async-prompt-dispatch.md) | Async prompt dispatch as a `kind:"prompt"` leaf sharing the `/turn` core | Proposed | -| [0029](0029-turn-sse-streaming.md) | Streaming `/turn` responses as an SSE representation via content negotiation | Proposed | +| # | Decision | Status | +| ----------------------------------------------------- | ------------------------------------------------------------------------------------------- | ----------- | +| [0001](0001-redis-session-backend.md) | Persist Pi session state through a pluggable Redis backend | Implemented | +| [0002](0002-k8s-sandbox-client-remote-exec.md) | Route Pi tool execution to a remote pod via `kubectl exec` | Implemented | +| [0003](0003-persistent-in-pod-channel.md) | Reuse a long-lived in-pod bash session for fast tool ops | Implemented | +| [0004](0004-knative-serverless-wrapper.md) | Expose the harness as a scale-to-zero Knative HTTP service | Implemented | +| [0005](0005-mcp-code-mode.md) | Originate MCP calls as code the model runs in the sandbox | Accepted | +| [0006](0006-generalized-credentialed-egress.md) | Placeholder-swap over a forward proxy for all credentialed egress | Accepted | +| [0007](0007-compaction-checkpoint-fast-path.md) | Ride Pi's native compaction entry as the resume checkpoint | Implemented | +| [0008](0008-experiments-harness.md) | Measure the loader as in-process reconstruction cost, not end-to-end latency | Implemented | +| [0009](0009-cluster-experiments.md) | Drive cluster experiments with bash extending `smoke.sh` | Implemented | +| [0010](0010-identity-spine.md) | Per-session SPIFFE identity, user in the attested path, minted by the orchestrator | Accepted | +| [0011](0011-harness-lockdown.md) | Defend the harness by defanging local execution, not by mediating egress | Accepted | +| [0012](0012-inference-injector.md) | A separate injector pod holds the provider key and owns public LLM egress | Accepted | +| [0013](0013-leaf-session-backend-reprioritization.md) | Reprioritize the harness as a leaf-session backend; defer the heavy credential plane | Accepted | +| [0014](0014-mvp-leaf-session-contract.md) | Prove the backend with a run-to-completion `/runs` invocation contract | Implemented | +| [0015](0015-async-leaf-completion.md) | KEDA ScaledJob over a Redis Streams queue for background leaf execution | Implemented | +| [0016](0016-human-gate.md) | Human-gate as a structured terminal plus externally-triggered continuation | Implemented | +| [0017](0017-registry-securitycontext-hardening.md) | Apply the non-root least-privilege securityContext baseline to agent pods | Implemented | +| [0018](0018-scheduled-leaf-dispatch.md) | Native Kubernetes CronJob as the scheduled dispatch start signal | Implemented | +| [0019](0019-ocp-fs-free-deployment.md) | Durable RWO EBS Sandbox CR, non-root under nonroot-v2, for the OCP deployment | Implemented | +| [0020](0020-fs-free-harness.md) | Filesystem-free harness (envelope inline + Redis, working set on Sandbox CR) | Implemented | +| [0021](0021-shared-sandbox-pool.md) | Shared sandbox pool via N Sandbox CRs with harness-side Redis-lease routing | Implemented | +| [0022](0022-workload-parameterized-sandbox-load.md) | Report the sharing ratio as an N-vs-workload curve, not a single number | Implemented | +| [0023](0023-sandbox-sharing-ratio-experiments.md) | Measure sandbox sharing capacity on runc; split Kata isolation into P4 | Implemented | +| [0024](0024-sandbox-transport-remote-exec.md) | Remote sandbox exec over a worker-dialed gRPC stream, contract as language-neutral Protobuf | Accepted | +| [0025](0025-authbridge-deployment-topology.md) | AuthBridge topology: shared LLM-egress gateway (AB1) + per-sandbox egress proxy (AB2) | Accepted | +| [0026](0026-rc1-static-inject-plugin.md) | RC1 credential injection via a dedicated `static-inject` plugin (not a broker service) | Accepted | +| [0027](0027-rc1-control-gate-and-hop2-realization.md) | RC1 control gate as IBAC-only; Hop-2 egress interception over plain HTTP | Accepted | +| [0028](0028-async-prompt-dispatch.md) | Async prompt dispatch as a `kind:"prompt"` leaf sharing the `/turn` core | Proposed | +| [0029](0029-turn-sse-streaming.md) | Streaming `/turn` responses as an SSE representation via content negotiation | Proposed | ## What an ADR is (and isn't) -| | ADR | Spec (`../specs/`) | Plan (`../plans/`) | -|---|---|---|---| -| Answers | *what we decided & why* | *what & why, in depth* (alternatives, trade-offs, deferred) | *how, in what order* | -| Size | short (one decision) | long (a whole design) | a checklist | -| Retention | **permanent, immutable** | committed, point-in-time | **local-only, ephemeral** | -| On change | write a **new** ADR that supersedes | add a `Status:` header, don't rewrite | delete once coded | +| | ADR | Spec (`../specs/`) | Plan (`../plans/`) | +| --------- | ----------------------------------- | ----------------------------------------------------------- | ------------------------- | +| Answers | _what we decided & why_ | _what & why, in depth_ (alternatives, trade-offs, deferred) | _how, in what order_ | +| Size | short (one decision) | long (a whole design) | a checklist | +| Retention | **permanent, immutable** | committed, point-in-time | **local-only, ephemeral** | +| On change | write a **new** ADR that supersedes | add a `Status:` header, don't rewrite | delete once coded | -An ADR is deliberately small: it records the *decision*, not the *design*. The full design +An ADR is deliberately small: it records the _decision_, not the _design_. The full design lives in a dated spec under [`../specs/`](../specs/); the ADR links to it. Use an ADR when a choice is (a) hard to reverse, (b) cross-cutting, or (c) likely to be questioned later ("why Connect instead of raw gRPC?", "why is the harness filesystem-free?"). @@ -59,7 +59,7 @@ Connect instead of raw gRPC?", "why is the harness filesystem-free?"). 1. **Immutable.** Once accepted, an ADR is never edited except to change its `Status` line (e.g. to `Superseded by ADR-0007`). To change a decision, write a **new** ADR that - references and supersedes the old one. The record of *why we once thought otherwise* has + references and supersedes the old one. The record of _why we once thought otherwise_ has value. 2. **Numbered, monotonic.** Files are `NNNN-kebab-title.md`, zero-padded, next free number. Numbers are never reused. @@ -80,4 +80,4 @@ cp docs/adrs/0000-adr-template.md docs/adrs/NNNN-your-decision.md --- -*Assisted-By: Claude (Anthropic AI) * +_Assisted-By: Claude (Anthropic AI) _ diff --git a/docs/demos/README.md b/docs/demos/README.md index 9ea73fa..7f33b11 100644 --- a/docs/demos/README.md +++ b/docs/demos/README.md @@ -1,30 +1,30 @@ # Demos — guided walkthroughs Hands-on tours you **drive by hand**, one command at a time, explaining as you go. Each one shows -something a conventional setup cannot do, and is structured in *acts* so it survives being +something a conventional setup cannot do, and is structured in _acts_ so it survives being performed live in front of an audience. ## What lives here -| Demo | Shows | Time | -|------|-------|------| -| [`serverless-harness-demo.md`](./serverless-harness-demo.md) | An agent that **scales to a true zero**, resumes from cold with full memory, then **fans out into a worker fleet** that appears on demand and vanishes when the queue drains | ~10 min | -| [`remote-sandbox-demo.md`](./remote-sandbox-demo.md) | A **sandbox outside the cluster** with zero inbound rules, executing a leaf's tool calls — one free-form prompt that names a different OS on each backend, and a secret planted by hand that the cluster reads back | ~10 min | +| Demo | Shows | Time | +| ------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | +| [`serverless-harness-demo.md`](./serverless-harness-demo.md) | An agent that **scales to a true zero**, resumes from cold with full memory, then **fans out into a worker fleet** that appears on demand and vanishes when the queue drains | ~10 min | +| [`remote-sandbox-demo.md`](./remote-sandbox-demo.md) | A **sandbox outside the cluster** with zero inbound rules, executing a leaf's tool calls — one free-form prompt that names a different OS on each backend, and a secret planted by hand that the cluster reads back | ~10 min | ## Demo vs. smoke test vs. spec These three overlap and are easy to confuse: -- **A demo (here)** is *performed*. It optimizes for a human explaining a claim out loud, so it +- **A demo (here)** is _performed_. It optimizes for a human explaining a claim out loud, so it narrates why each step matters and deliberately pauses on the moments that are hard to believe. Every command is copy-pasteable. -- **A smoke test** ([`../../deploy/knative/SMOKE.md`](../../deploy/knative/SMOKE.md)) is *asserted*. +- **A smoke test** ([`../../deploy/knative/SMOKE.md`](../../deploy/knative/SMOKE.md)) is _asserted_. It optimizes for an unattended pass/fail with no narration. Most demos here have a scripted sibling — `remote-sandbox-demo.md` is `make demo-remote-sandbox`, and Act 2 of `serverless-harness-demo.md` is `leaf-async-smoke.sh`. Prefer the script when you want a pass/fail; - prefer the demo when you want to *convince someone*. -- **A spec or ADR** ([`../specs/`](../specs/), [`../adrs/`](../adrs/)) records the ***why*** — the - decision and its rejected alternatives. A demo shows the *what*, and goes stale when the + prefer the demo when you want to _convince someone_. +- **A spec or ADR** ([`../specs/`](../specs/), [`../adrs/`](../adrs/)) records the _**why**_ — the + decision and its rejected alternatives. A demo shows the _what_, and goes stale when the commands change; a decision record does not. ## Conventions @@ -35,11 +35,11 @@ If you add a demo, follow the shape of the two above: should know in fifteen seconds whether this demo is the one they want. - **Acts, with lettered sub-steps** (`### 1a.`, `### 1b.`) so you can resume mid-performance and so a reviewer can cite a step. -- **Blockquote callouts (`>`) for the narration** — what to *say*, and which traps the step defends +- **Blockquote callouts (`>`) for the narration** — what to _say_, and which traps the step defends against. Keep them out of the code blocks so the commands stay copy-pasteable. - **Show expected output** inline. A demo whose output you cannot compare against is a demo that has silently rotted. - **End with "What just happened"** (recap the claims, numbered) and **"Cleanup"** (leave the - cluster as you found it — restore any env you flipped *first*). -- **Be honest about limits.** A closing "Notes and limits" section naming what the demo does *not* + cluster as you found it — restore any env you flipped _first_). +- **Be honest about limits.** A closing "Notes and limits" section naming what the demo does _not_ show is worth more than an extra act, because it is what stops someone over-promising in a room. diff --git a/docs/demos/remote-sandbox-demo.md b/docs/demos/remote-sandbox-demo.md index e9bb8ab..a95d109 100644 --- a/docs/demos/remote-sandbox-demo.md +++ b/docs/demos/remote-sandbox-demo.md @@ -4,8 +4,8 @@ A ~10-minute walkthrough of **SandboxTransport**: the harness dispatches a leaf' sandbox running as a plain `docker run` **on your laptop** — outside the cluster, with **zero inbound rules**, holding no cluster credential. -The task — read a file and say what it contains — is just a vehicle. The real show is *which -machine's filesystem answers*. You will send the same free-form prompt twice and watch the model +The task — read a file and say what it contains — is just a vehicle. The real show is _which +machine's filesystem answers_. You will send the same free-form prompt twice and watch the model name a different OS each time, then plant a secret in a container by hand and watch the cluster read it back. @@ -20,11 +20,11 @@ laptop Neither address is inbound to the laptop. -| Act | What a normal remote sandbox needs | What SandboxTransport needs | -|-----|-----------------------------------|-----------------------------| -| **1 — Inverted connectivity** | An inbound port, a firewall rule, a public address the cluster can reach | **Nothing.** The worker dials *out* and parks a stream. `docker run` with no `-p` at all | -| **2 — Provable placement** | Trust that the config routed where you think | A **fingerprint** and a **structural guard** — a green run on the wrong backend is made impossible | -| **3 — Zero standing authority** | A kubeconfig, or an agent with cluster reach | One bearer token. No LLM key, no kubeconfig, no orchestration | +| Act | What a normal remote sandbox needs | What SandboxTransport needs | +| ------------------------------- | ------------------------------------------------------------------------ | -------------------------------------------------------------------------------------------------- | +| **1 — Inverted connectivity** | An inbound port, a firewall rule, a public address the cluster can reach | **Nothing.** The worker dials _out_ and parks a stream. `docker run` with no `-p` at all | +| **2 — Provable placement** | Trust that the config routed where you think | A **fingerprint** and a **structural guard** — a green run on the wrong backend is made impossible | +| **3 — Zero standing authority** | A kubeconfig, or an agent with cluster reach | One bearer token. No LLM key, no kubeconfig, no orchestration | Prefer it non-interactive? `make demo-remote-sandbox` does all of this in one command and asserts every step. This document is the version you drive by hand so you can explain each move. @@ -91,7 +91,7 @@ curl -s -o /dev/null -w 'harness HTTP %{http_code}\n' --max-time 5 -H "$HOSTHDR" > **`404` is success.** This is a transport check, not a health check: any response proves the > tunnel and Host header reach the harness. Skipping it is a trap — an empty `/runs` reply later -> is indistinguishable from an unreachable *model*, and the two have completely different fixes. +> is indistinguishable from an unreachable _model_, and the two have completely different fixes. ### Build the worker image @@ -107,8 +107,8 @@ docker build --load -f remote-worker/Dockerfile -t dev.local/remote-worker:demo # Act 1: A sandbox with no inbound route -**The claim a normal remote sandbox can't make:** *nothing can reach me, and I am still serving -your cluster's tool calls.* +**The claim a normal remote sandbox can't make:** _nothing can reach me, and I am still serving +your cluster's tool calls._ ### 1a. Bring up the relay @@ -118,7 +118,7 @@ kubectl -n $NS rollout status deploy/sandbox-relay --timeout=90s ``` > The relay is the only thing the worker will dial. It is **inert** until both a worker attaches -> *and* the harness is switched to the remote path — so nothing is routed anywhere yet. +> _and_ the harness is switched to the remote path — so nothing is routed anywhere yet. ### 1b. Generate the registration token @@ -130,7 +130,7 @@ kubectl -n $NS rollout status deploy/sandbox-relay --timeout=90s > Relay auth is **fail-closed**: a token mismatch rejects the Attach before the stream is ever > parked. We mint a fresh token per run rather than using `relay-deployment.yaml`'s `dev-token`, -> because that value is a repo constant and therefore public. Patch *before* waiting on the +> because that value is a repo constant and therefore public. Patch _before_ waiting on the > rollout, so the pod that becomes Ready is already the one holding this token. ### 1c. Open the tunnel — and prove the relay is really serving @@ -145,7 +145,7 @@ grep -qE 'error forwarding|connection refused|lost connection' /tmp/demo-remote/ ``` > **A bare TCP connect is not enough.** `kubectl port-forward` accepts your local connection -> first and only *then* tries the pod, so a dead relay still gives you a successful connect. +> first and only _then_ tries the pod, so a dead relay still gives you a successful connect. > Reading the forward log is what separates "the relay is dead" from "a container can't route > here" — two failures with completely different fixes. The relay also needs ~4s after `Running` > to bind, because `node --import tsx` compiles its TypeScript at startup; probing immediately is @@ -173,7 +173,7 @@ docker inspect sh-demo-remote-worker --format '{{range .Config.Env}}{{println .} > A bearer token, a sandbox id, a relay address. **No LLM key, no kubeconfig, no > orchestration.** If this container is stolen, the attacker gets a scoped token to one relay. -### 1e. Registration *is* the live stream +### 1e. Registration _is_ the live stream **Look at T1** — the record just appeared: @@ -183,14 +183,14 @@ sbx-laptop-demo ``` > Nothing polled. Nothing heartbeated a URL. Redis holds this record **only while the Attach -> stream is open** — the registration *is* the stream. `"transport":"grpc"` is how the harness +> stream is open** — the registration _is_ the stream. `"transport":"grpc"` is how the harness > knows to route over the relay instead of `kubectl exec`. We come back to T1 in Act 3. --- # Act 2: Prove which machine ran the command -**The claim a config change can't make on its own:** *the exec provably ran there, not here.* +**The claim a config change can't make on its own:** _the exec provably ran there, not here._ ### 2a. Establish the discriminator first @@ -204,10 +204,10 @@ PRETTY_NAME="Alpine Linux v3.20" <- in-cluster pool PRETTY_NAME="Red Hat Enterprise Linux 9.8 (Plow)" <- remote host container ``` -> The pool is Alpine, the worker is RHEL. So one free-form question — *what does -> `/etc/os-release` say?* — gets a different answer depending on which machine ran it, and the +> The pool is Alpine, the worker is RHEL. So one free-form question — _what does +> `/etc/os-release` say?_ — gets a different answer depending on which machine ran it, and the > model **names the OS it read** rather than handing you a flag you have to trust. Verify the -> fingerprint *before* anything relies on it; an unverified discriminator makes every later +> fingerprint _before_ anything relies on it; an unverified discriminator makes every later > assertion meaningless. Set the prompt you will send unchanged to both backends: @@ -241,9 +241,9 @@ The file /etc/os-release reports Alpine Linux v3.20. > > `kind:"prompt"` is what makes this a free-form leaf: the reply comes back as `.text`, the > model's own words, with `status: "responded"`. No verdict schema, no `submit_verdict` tool — -> which is why the answer can *name* what it read. +> which is why the answer can _name_ what it read. -### 2c. Flip to the remote path — and make a pod win *impossible* +### 2c. Flip to the remote path — and make a pod win _impossible_ Snapshot the env first. **Look at what has to survive the flip:** @@ -282,7 +282,7 @@ kubectl get pods -n $NS -l sh.kagenti.io/sandbox-pool=demo-remote-only \ # => 0 ``` -> **Say this carefully.** `SH_REMOTE_SANDBOX=1` *alone does not route to the worker.* +> **Say this carefully.** `SH_REMOTE_SANDBOX=1` _alone does not route to the worker._ > `select-sandbox.ts` builds `candidates = [...pods, ...grpcRecs]` and leases least-loaded-first, > so an idle in-cluster pod can still win the lease — and you would get a green demo that proved > nothing. Pointing the selector at a label **no pod carries** means a pod cannot win a lease it @@ -340,23 +340,23 @@ BEFORE (8) AFTER (10) **The property that matters:** untouched entries are passed through as **whole objects**, never reconstructed — which is what preserves those three `secretKeyRef`s. `kubectl set env` does not -work on a Knative `Service` at all (*"no kind Service is registered"*), and anything that rebuilds +work on a Knative `Service` at all (_"no kind Service is registered"_), and anything that rebuilds the array from name/value pairs flattens `valueFrom` to an empty string. The model call then fails looking exactly like an unreachable endpoint. Order shifts, which is harmless: Kubernetes only cares about env ordering for `$(VAR)` interpolation, which this env does not use. **The kubectl half — replace the whole array.** JSON Patch has no "upsert by name": array ops -address elements by *index*, and indices shift as you add and remove. Computing the final array in +address elements by _index_, and indices shift as you add and remove. Computing the final array in jq and replacing `/spec/template/spec/containers/0/env` once sidesteps that arithmetic and lands atomically. `containers/0` is the first (user) container in the revision template. **And what the three values do:** -| Var | Effect | -|-----|--------| -| `SH_REMOTE_SANDBOX=1` | enables the remote-sandbox code path at all | -| `SH_RELAY_ADDR=sandbox-relay.default.svc:8443` | where the *harness* dials the relay — in-cluster DNS, the other end of the worker's outbound tunnel | -| `KAGENTI_SANDBOX_POOL_SELECTOR=…pool=demo-remote-only` | a label no pod carries, so the pod candidate set is empty | +| Var | Effect | +| ------------------------------------------------------ | --------------------------------------------------------------------------------------------------- | +| `SH_REMOTE_SANDBOX=1` | enables the remote-sandbox code path at all | +| `SH_RELAY_ADDR=sandbox-relay.default.svc:8443` | where the _harness_ dials the relay — in-cluster DNS, the other end of the worker's outbound tunnel | +| `KAGENTI_SANDBOX_POOL_SELECTOR=…pool=demo-remote-only` | a label no pod carries, so the pod candidate set is empty | The third is the load-bearing one for the demo's honesty. The first two alone would leave idle Alpine pods in the candidate set, and least-loaded-first could hand the exec to one. @@ -379,7 +379,7 @@ The file /etc/os-release reports Red Hat Enterprise Linux 9.8 (Plow). > Same prompt as 2b, **different OS named.** Now check it in both directions — the reply must say > `Red Hat` **and** must not say `Alpine`. Assert against the reply you already captured; a second -> `curl` with the same `sessionId` would *resume* that session rather than ask afresh: +> `curl` with the same `sessionId` would _resume_ that session rather than ask afresh: ```bash grep -qi 'red hat' <<<"$REPLY" && echo "ok: named Red Hat" || echo "FAIL: did not name Red Hat" @@ -387,7 +387,7 @@ grep -qi 'alpine' <<<"$REPLY" && echo "FAIL: named Alpine" || echo "ok: did not ``` > Both directions on purpose: a free-form reply has no flag to flip, so "it landed on an Alpine -> pod" is ruled out by asserting the OS it must *not* have read is absent too. This is the one +> pod" is ruled out by asserting the OS it must _not_ have read is absent too. This is the one > place the free-form version is weaker than a `CLEAR`/`FLAGGED` verdict — a reply that mentions > neither OS fails the first check rather than being caught as nonsense. Act 3 is what closes > that gap, and it is the stronger proof anyway. @@ -396,8 +396,8 @@ grep -qi 'alpine' <<<"$REPLY" && echo "FAIL: named Alpine" || echo "ok: did not # Act 3: The closer — plant a secret, watch the cluster read it back -**The claim that ends the argument:** *you created this evidence thirty seconds ago, on this -laptop, and the cluster just read it.* +**The claim that ends the argument:** _you created this evidence thirty seconds ago, on this +laptop, and the cluster just read it._ ### 3a. Write a marker only your laptop has @@ -426,7 +426,7 @@ The file /tmp/proof.txt contains the marker string: tuscan-lentils-29765 ``` > **Note what the free-form reply buys you here.** It does not confirm a string you already -> supplied — it *reads one back to you*. Check it against the `$MARK` you printed thirty seconds +> supplied — it _reads one back to you_. Check it against the `$MARK` you printed thirty seconds > ago: ```bash @@ -435,7 +435,7 @@ grep -qF "$MARK" <<<"$PROOF" \ ``` > There is no `kubectl exec` anywhere in that path, no inbound route to this machine, and the -> worker holds no cluster credential — only a token it used to dial *out*. An answer about +> worker holds no cluster credential — only a token it used to dial _out_. An answer about > `/etc/os-release` can be argued with — image drift, a lucky guess from context. A random string > you generated yourself, echoed back verbatim, cannot be. @@ -448,10 +448,10 @@ docker stop sh-demo-remote-worker **Watch T1.** The record clears on its own: > Nothing deleted it. The Attach stream closed and the record went with it. That is what -> "registration *is* the live stream" means — and why the harness never routes to a sandbox that +> "registration _is_ the live stream" means — and why the harness never routes to a sandbox that > has quietly gone away. -> `make demo-remote-sandbox` asserts this act too — the planted marker *and* this teardown. The +> `make demo-remote-sandbox` asserts this act too — the planted marker _and_ this teardown. The > one exception is `--keep`, which promises the worker is still running when the run ends: proving > the record clears means closing the stream, so the script skips this step and tells you to do it > by hand instead. @@ -464,7 +464,7 @@ You drove a sandbox that: 1. **Had no inbound route** — `docker run` with no `-p`, no firewall rule, reachable by nothing (Act 1d). -2. **Registered by existing** — its presence record *was* its open stream, and vanished with it +2. **Registered by existing** — its presence record _was_ its open stream, and vanished with it (Act 1e, 3c). 3. **Provably ran the exec** — same prompt, and the model named a different OS each time, with a pool selector that made a pod win structurally impossible (Act 2). @@ -524,8 +524,8 @@ make demo-remote-sandbox-teardown # asks before deleting the cluster; DEMO_ARG probes for this and escalates automatically, with a warning. - **Live streaming, abort mid-stream, dual-ended timeout and reconnect→dedup** are implemented and unit-tested but not shown here — tracked in - [#198](https://github.com/rossoctl/serverless-harness/issues/198). The honest line: *the - transport does it, this demo doesn't show it yet.* + [#198](https://github.com/rossoctl/serverless-harness/issues/198). The honest line: _the + transport does it, this demo doesn't show it yet._ Reference: [`../../deploy/knative/README-worker.md`](../../deploy/knative/README-worker.md) §"Laptop demo" and [`../../deploy/knative/demo-remote-worker.sh`](../../deploy/knative/demo-remote-worker.sh). diff --git a/docs/demos/serverless-harness-demo.md b/docs/demos/serverless-harness-demo.md index dc5aee2..9ca8937 100644 --- a/docs/demos/serverless-harness-demo.md +++ b/docs/demos/serverless-harness-demo.md @@ -1,7 +1,7 @@ # Demo: "The agent that isn't there" A 10-minute walkthrough that shows what a **serverless** AI agent does that a -normal always-on agent *can't*. The task — a small security review of a repo — +normal always-on agent _can't_. The task — a small security review of a repo — is just a vehicle. The real show is in your pod-watch pane: watch the agent **cold-start from zero, drop back to zero, resume with full memory, and then fan out into a fleet of worker pods that appear on demand and vanish when the @@ -9,10 +9,10 @@ work drains.** Two differentiators, two acts: -| Act | What a plain agent does | What the harness does | -|-----|-------------------------|-----------------------| -| **1 — Durable resume** | Stays resident (burning compute) or forgets on restart | Scales to **zero**, then cold-starts and **remembers** — state lives in Redis, not the process | -| **2 — Fan-out from zero** | Grinds a batch serially in one resident process | Materializes **N worker pods on demand**, drains the queue, collapses back to **zero** | +| Act | What a plain agent does | What the harness does | +| ------------------------- | ------------------------------------------------------ | ---------------------------------------------------------------------------------------------- | +| **1 — Durable resume** | Stays resident (burning compute) or forgets on restart | Scales to **zero**, then cold-starts and **remembers** — state lives in Redis, not the process | +| **2 — Fan-out from zero** | Grinds a batch serially in one resident process | Materializes **N worker pods on demand**, drains the queue, collapses back to **zero** | --- @@ -69,8 +69,8 @@ export BASE="http://localhost:8080" # Act 1: Durable resume across a true zero -**The claim a plain agent can't make:** *I cost nothing while idle, and I still -remember everything.* +**The claim a plain agent can't make:** _I cost nothing while idle, and I still +remember everything._ ### 1a. Confirm you're at zero @@ -126,14 +126,14 @@ Watch **T1** cold-start a **fresh** pod that answers correctly — it names > A brand-new pod with no memory of its own just recalled the policy. The state > survived the trip to zero in Redis. That's **durable resume across a cold -> start** — scale-to-zero economics *without* amnesia. +> start** — scale-to-zero economics _without_ amnesia. --- # Act 2: A fleet from zero -**The claim a plain agent can't make:** *I run your batch in parallel by -conjuring workers on demand, then I disappear.* +**The claim a plain agent can't make:** _I run your batch in parallel by +conjuring workers on demand, then I disappear._ We'll review five files in the repo at once. Each file becomes an independent **async leaf**: the harness accepts it instantly (`202`), pushes it onto a Redis diff --git a/docs/executive-overview-leaf-session.md b/docs/executive-overview-leaf-session.md index f537294..7c520ea 100644 --- a/docs/executive-overview-leaf-session.md +++ b/docs/executive-overview-leaf-session.md @@ -1,7 +1,7 @@ # Leaf-Session Backend — Executive Overview -**Status:** MVP complete (Phase 1 + Leaf-Session shipped). Zero-trust deferred to Phase 2. -**Repo:** `kagenti/serverless-harness` (private) +**Status:** MVP complete (Phase 1 + Leaf-Session shipped). Zero-trust deferred to Phase 2. +**Repo:** `kagenti/serverless-harness` (private) **Date:** 2026-06-28 --- @@ -63,25 +63,25 @@ The leaf-session backend supports three dispatch patterns for running AI agent " ### Key Components -| Component | Role | -|-----------|------| -| **Knative Service** | Scale-to-zero HTTP endpoint; handles sync `/runs` and enqueues async work | -| **Redis** | Session persistence (durable resume by `sessionId`), work queue (Streams), gate state | -| **KEDA ScaledJob** | Autoscales worker pods 0→10 based on `lagCount` + `pendingEntriesCount` | -| **sandbox-0** | Isolated execution pod; harness routes tool calls here via `kubectl exec` (brain/hands split) | -| **Shared PVC** | Volume-envelope contract — inputs, results, and markers travel as files, not HTTP bodies | -| **CronJob** | Archetype C scheduler; fires `cron-dispatch` on cron schedule | +| Component | Role | +| ------------------- | --------------------------------------------------------------------------------------------- | +| **Knative Service** | Scale-to-zero HTTP endpoint; handles sync `/runs` and enqueues async work | +| **Redis** | Session persistence (durable resume by `sessionId`), work queue (Streams), gate state | +| **KEDA ScaledJob** | Autoscales worker pods 0→10 based on `lagCount` + `pendingEntriesCount` | +| **sandbox-0** | Isolated execution pod; harness routes tool calls here via `kubectl exec` (brain/hands split) | +| **Shared PVC** | Volume-envelope contract — inputs, results, and markers travel as files, not HTTP bodies | +| **CronJob** | Archetype C scheduler; fires `cron-dispatch` on cron schedule | ### Single Image, Two Entry Points The Knative Service and leaf-worker are the **same container image** (`serverless-harness`) with different entry points: -| | Knative Service | leaf-worker (KEDA ScaledJob) | -|---|---|---| -| **Entry point** | `server.ts` — HTTP server | `leaf-job.ts` — queue drain loop | -| **Triggered by** | HTTP request (`POST /runs`) | Redis Streams queue depth | -| **Calls** | `runLeaf()` inline (sync) or enqueues to Redis (async) | `processOne()` → `runLeaf()` | -| **kubectl exec → sandbox-0** | Yes | Yes | +| | Knative Service | leaf-worker (KEDA ScaledJob) | +| ---------------------------- | ------------------------------------------------------ | -------------------------------- | +| **Entry point** | `server.ts` — HTTP server | `leaf-job.ts` — queue drain loop | +| **Triggered by** | HTTP request (`POST /runs`) | Redis Streams queue depth | +| **Calls** | `runLeaf()` inline (sync) or enqueues to Redis (async) | `processOne()` → `runLeaf()` | +| **kubectl exec → sandbox-0** | Yes | Yes | Both paths converge on `runLeaf()`, which uses `K8sSandboxClient` to route all tool execution into sandbox-0. The "brain" (model inference + session logic) runs in whichever pod called `runLeaf()`; the "hands" (actual code/tool execution) always run in sandbox-0. @@ -102,17 +102,18 @@ POST /runs { sessionId, inputsRef, resultRef, async: true } → async (202 Acc ### Cluster Footprint at Rest -| Always on | Scales to zero | -|-----------|----------------| -| **Redis** (session state + queue) | Knative Service (cold-starts on first request, sub-second) | -| **sandbox-0** (persistent workspace) | leaf-worker (KEDA, zero when queue empty) | -| | cron-dispatch (exists only during CronJob fire) | +| Always on | Scales to zero | +| ------------------------------------ | ---------------------------------------------------------- | +| **Redis** (session state + queue) | Knative Service (cold-starts on first request, sub-second) | +| **sandbox-0** (persistent workspace) | leaf-worker (KEDA, zero when queue empty) | +| | cron-dispatch (exists only during CronJob fire) | **2 pods at idle.** All compute (Knative Service, leaf-workers) scales to zero when no work is pending. Even the async enqueue path cold-starts from zero — the Knative activator intercepts the first request, spins up a pod, the pod enqueues and returns 202, then idles back to zero after 30s. sandbox-0 stays up because it holds the persistent working directory (files, packages, git state) that must survive across leaf invocations. ### Security Posture (PR #16) All workload pods run hardened: + - Non-root UID 65532, `fsGroup: 65532` for PVC group-write - `readOnlyRootFilesystem: true` + `/tmp` emptyDir for scratch - `capabilities: { drop: [ALL] }`, seccomp `RuntimeDefault` @@ -140,6 +141,7 @@ A single demonstration exercising all three archetypes in sequence: ### 3.2 MVP Boundary **What works today (no zero-trust required):** + - All three dispatch archetypes on Kind and OpenShift - Scale-to-zero with sub-second cold-start resume - Durable sessions surviving pod eviction @@ -147,6 +149,7 @@ A single demonstration exercising all three archetypes in sequence: - Hardened security posture (non-root, read-only rootfs, drop caps) **Known limitations (deferred to Phase 2):** + - No credential injection — `ANTHROPIC_API_KEY` is a pre-provisioned K8s Secret (trust-the-operator) - No egress policy — sandbox can reach any endpoint (no NetworkPolicy enforcement) - No per-session identity — all leaves share the service account's SPIFFE identity @@ -156,16 +159,16 @@ A single demonstration exercising all three archetypes in sequence: ## Phase 2 Preview (Z-track, design complete) -| Milestone | Adds | -|-----------|------| -| Z1 Identity Spine | Per-session SPIFFE SVID via SPIRE | -| Z2 Harness Lock-Down | Secret-free distroless container, default-deny egress | -| Z3 Inference Injector | Provider-key chokepoint, mTLS to LLM gateway | -| Z4 MCP Code-Mode | Model-authored code runs in sandbox, transparent AuthBridge | -| Z5 Credentialed Egress | Forward proxy + baked CA for sandbox outbound | -| Z6 Subagents | Isolated child sessions with scoped credentials | -| Z7 Validation | Red-team + formal verification of the credential plane | +| Milestone | Adds | +| ---------------------- | ----------------------------------------------------------- | +| Z1 Identity Spine | Per-session SPIFFE SVID via SPIRE | +| Z2 Harness Lock-Down | Secret-free distroless container, default-deny egress | +| Z3 Inference Injector | Provider-key chokepoint, mTLS to LLM gateway | +| Z4 MCP Code-Mode | Model-authored code runs in sandbox, transparent AuthBridge | +| Z5 Credentialed Egress | Forward proxy + baked CA for sandbox outbound | +| Z6 Subagents | Isolated child sessions with scoped credentials | +| Z7 Validation | Red-team + formal verification of the credential plane | --- -*Assisted-By: Claude Code* +_Assisted-By: Claude Code_ diff --git a/docs/experiment-results.md b/docs/experiment-results.md index cfb9cda..cbbae8c 100644 --- a/docs/experiment-results.md +++ b/docs/experiment-results.md @@ -1,7 +1,7 @@ # Serverless Harness — Experiment Results (E1–E5) -*Consolidated findings, June 2026. Source data: `experiments/RESULTS.md` (E2, E5), -`deploy/knative/EXPERIMENTS.md` (E1, E3, E4). Designs: `docs/specs/2026-06-{23,24,25}-*.md`.* +_Consolidated findings, June 2026. Source data: `experiments/RESULTS.md` (E2, E5), +`deploy/knative/EXPERIMENTS.md` (E1, E3, E4). Designs: `docs/specs/2026-06-{23,24,25}-*.md`._ ## The thesis under test @@ -10,13 +10,13 @@ The serverless harness runs a [Pi](https://github.com/earendil-works) coding age **remote sandbox**, so a session is durable infrastructure rather than a long-lived process. The five experiments test whether that architecture pays off without breaking correctness: -| # | Claim | Verdict | Headline result | -|---|-------|---------|-----------------| -| **E1** | Scale-to-zero is cheaper than always-on for idle-heavy use | ✅ PASS | serverless **0.25×** the pod-seconds (~75% cheaper) | -| **E2** | Compaction-checkpoint keeps cold-start reconstruction O(tail), not O(total) | ✅ PASS | checkpoint reads **constant 6 entries** vs backend's 53→5003; ratio **8.8→833.8** | -| **E3** | A session is portable: a fresh instance reconstructs equivalent context from the log | ✅ PASS | fidelity (≡ full replay) + mobility (fresh pod recalled the planted token) | -| **E4** | Crash recovery is a byproduct of the externalized log | ✅ PASS | pod force-killed mid-session → next turn recovered all completed turns | -| **E5** | Per-turn token spend can be capped and enforced | ✅ PASS | tool call blocked + exactly one `abort` past the cap; inert when unset | +| # | Claim | Verdict | Headline result | +| ------ | ------------------------------------------------------------------------------------ | ------- | --------------------------------------------------------------------------------- | +| **E1** | Scale-to-zero is cheaper than always-on for idle-heavy use | ✅ PASS | serverless **0.25×** the pod-seconds (~75% cheaper) | +| **E2** | Compaction-checkpoint keeps cold-start reconstruction O(tail), not O(total) | ✅ PASS | checkpoint reads **constant 6 entries** vs backend's 53→5003; ratio **8.8→833.8** | +| **E3** | A session is portable: a fresh instance reconstructs equivalent context from the log | ✅ PASS | fidelity (≡ full replay) + mobility (fresh pod recalled the planted token) | +| **E4** | Crash recovery is a byproduct of the externalized log | ✅ PASS | pod force-killed mid-session → next turn recovered all completed turns | +| **E5** | Per-turn token spend can be capped and enforced | ✅ PASS | tool call blocked + exactly one `abort` past the cap; inert when unset | All five pass. Together they validate the economic case (E1), the optimization that makes scale-to-zero practical at length (E2), the correctness guarantees that make it safe (E3, E4), @@ -36,18 +36,18 @@ A sampler polls running pods every 5s; **pod-seconds = Σ(running pods × 5s)** **Result.** `persistent=380s`, `serverless=95s`, **ratio 0.25** — serverless used a quarter of the pod-runtime (~75% cheaper). Gate: serverless ≤ 0.6 × persistent. **PASS.** -**Caveat / methodology.** Serverless pod-seconds are roughly *constant per turn* (cold-start + +**Caveat / methodology.** Serverless pod-seconds are roughly _constant per turn_ (cold-start + work + ~30s scale-to-zero retention), independent of idle length, while persistent grows with the window — so the saving widens with idle time. A short idle (≈120s) does **not** clear the gate (ratio ~0.78); the result reflects a genuinely idle-heavy pattern. pod-seconds is a runtime proxy, -not a billing model. *Reproduce:* `deploy/knative/e1-economics.sh`. +not a billing model. _Reproduce:_ `deploy/knative/e1-economics.sh`. ## E2 — Local reconstruction cost (the compaction-checkpoint fast path) **Claim.** On cold start, the M5 `openFromCheckpoint` loader reconstructs a session in **O(tail)** (read only the latest-compaction-forward slice) rather than **O(total)** (full-log replay). -**Honest framing.** Pi's native compaction already bounds the *LLM* context, so this is **not** an +**Honest framing.** Pi's native compaction already bounds the _LLM_ context, so this is **not** an LLM-latency win — it is a **local** cost win (Redis read volume, indexing, the leaf→root walk). E2 measures exactly that, in-process, with a counting backend. @@ -55,20 +55,21 @@ E2 measures exactly that, in-process, with a counting backend. reads a **constant 6 entries / ~900 bytes**, while `openFromBackend` reads the whole log (53→5003 entries, 7.5KB→707KB). The backend/checkpoint ratio rises **8.8 → 33.8 → 167.2 → 833.8**, and `buildSessionContext()` is byte-identical under both loaders at every N (correctness preserved). -**PASS.** *Reproduce:* `pnpm -C experiments test e2-reconstruction-cost`. +**PASS.** _Reproduce:_ `pnpm -C experiments test e2-reconstruction-cost`. ## E3 — Session fidelity + mobility -**Claim.** A session's state lives in the log, so any fresh instance can reconstruct *equivalent* +**Claim.** A session's state lives in the log, so any fresh instance can reconstruct _equivalent_ context — the basis for scale-to-zero and for moving a session between pods. **Two halves.** + - **Fidelity** (M5): the checkpoint-reconstructed `buildSessionContext()` deep-equals a full replay — proven by the M5 parity gate (`checkpoint.test.ts`) and re-confirmed at every N in E2. - **Mobility** (M7, cluster): plant a fact, force the pod to zero (confirmed gone), then a follow-up on a **fresh** pod recalled the planted token (`ZEBRA42`) from the Redis log. -**Result.** Both hold. **PASS.** *Reproduce:* `deploy/knative/e3-mobility.sh` (mobility); +**Result.** Both hold. **PASS.** _Reproduce:_ `deploy/knative/e3-mobility.sh` (mobility); `harness/test/checkpoint.test.ts` (fidelity). ## E4 — Crash recovery as a byproduct @@ -80,7 +81,7 @@ mid-session loses no committed work. --force` mid-session, then issue a recovery turn on the freshly-started pod. **Result.** The post-crash turn recalled all three planted facts (APPLE/BANANA/CHERRY) — zero -completed-turn loss. **PASS.** *Reproduce:* `deploy/knative/e4-recovery.sh`. +completed-turn loss. **PASS.** _Reproduce:_ `deploy/knative/e4-recovery.sh`. ## E5 — Budget-voter enforcement @@ -91,7 +92,7 @@ baseline) and blocks the next `tool_call` once over cap, appending one `abort` l **Result.** Over cap → tool call blocked and **exactly one** `abort` persisted to real Redis; cap unset → inert (no block, no `abort`). A key-gated live run confirms the same end-to-end with a -real model. **PASS.** *Reproduce:* `pnpm -C experiments test e5-budget-structural` (gate); +real model. **PASS.** _Reproduce:_ `pnpm -C experiments test e5-budget-structural` (gate); `e5-budget-live.test.ts` (live, `SH_RUN_LIVE=1`). --- @@ -106,7 +107,7 @@ real model. **PASS.** *Reproduce:* `pnpm -C experiments test e5-budget-structura surfaced three real defects that static review missed and that were then fixed at root cause: a `start_sampler` command-substitution hang, an E3 scale-to-zero fall-through (false-pass risk), and an orchestrator `set -e` abort that skipped result-writing. The passing results below were - produced *after* those fixes, on green re-runs. + produced _after_ those fixes, on green re-runs. ## How to reproduce everything @@ -128,4 +129,4 @@ cold-start reconstruction (E2)**, while **preserving session fidelity and mobili to the agent's own logic (Pi is forked only for the pluggable storage backend). This completes the pi-track plan's E1–E5 experiment set. -*Assisted-By: Claude (Anthropic AI) * +_Assisted-By: Claude (Anthropic AI) _ diff --git a/docs/notes/swebench-image-facts.md b/docs/notes/swebench-image-facts.md index b79d845..89a1544 100644 --- a/docs/notes/swebench-image-facts.md +++ b/docs/notes/swebench-image-facts.md @@ -17,7 +17,7 @@ probes. ### Evidence **Source (`swebench/harness/docker_build.py`, `build_env_images()` / -`build_instance_image()`):** env images are *always built locally* from a +`build_instance_image()`):** env images are _always built locally_ from a generated Dockerfile (`env_dockerfile` → `get_dockerfile_env`), never pulled. Only the instance image build path checks `test_spec.is_remote_image` (true when a `namespace` is passed to `make_test_spec`) and, if so, does @@ -49,10 +49,10 @@ Task 3 **cannot** pull a shared per-env image directly (it does not exist on the registry). Two ways to still get the shared conda env into the baked sandbox image; **recommended: option (a)**: -- **(a) Recommended — pull one representative *instance* image per env-key +- **(a) Recommended — pull one representative _instance_ image per env-key group, then extract the conda env from it.** Because `env_image_key` is empirically stable across every instance that shares the same `(repo, - version)` (see §3 empirical check — the *content* of the shared `testbed` +version)` (see §3 empirical check — the _content_ of the shared `testbed` conda env is identical across those instances even though only instance-level artifacts are published), pulling **any single** already-published instance image for a given env-key group and running `conda-pack` on its @@ -78,16 +78,16 @@ env image locally built with this exact key." All templates come from `swebench.harness.test_spec.test_spec.TestSpec` properties (`swebench/harness/test_spec/test_spec.py`). -| Image level | Template | Published on Docker Hub? | Notes | -|---|---|---|---| -| Base | `sweb.base.{ext}.{arch}[.{hash10}]:{tag}` | No | `{hash10}` only present if `docker_specs != {}`. | -| Env | `sweb.env.{ext}.{arch}.{hash22}:{tag}` | **No** (verdict §1) | `{hash22}` = first 22 hex chars of `sha256(str(env_script_list) [+ str(docker_specs)])`. No namespace ever prepended. | -| Instance | `{namespace}/sweb.eval.{arch}.{instance_id_lower}:{tag}` (namespace only when `namespace is not None`) | **Yes** | Default `namespace="swebench"` (see `swebench/harness/run_evaluation.py:285`). `instance_id` is lower-cased and every `__` is replaced with `_1776_` (SWE-bench's escape for the org/repo separator, since Docker repo names disallow `__`). | +| Image level | Template | Published on Docker Hub? | Notes | +| ----------- | ------------------------------------------------------------------------------------------------------ | ------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| Base | `sweb.base.{ext}.{arch}[.{hash10}]:{tag}` | No | `{hash10}` only present if `docker_specs != {}`. | +| Env | `sweb.env.{ext}.{arch}.{hash22}:{tag}` | **No** (verdict §1) | `{hash22}` = first 22 hex chars of `sha256(str(env_script_list) [+ str(docker_specs)])`. No namespace ever prepended. | +| Instance | `{namespace}/sweb.eval.{arch}.{instance_id_lower}:{tag}` (namespace only when `namespace is not None`) | **Yes** | Default `namespace="swebench"` (see `swebench/harness/run_evaluation.py:285`). `instance_id` is lower-cased and every `__` is replaced with `_1776_` (SWE-bench's escape for the org/repo separator, since Docker repo names disallow `__`). | - **Tag:** `latest` (the `LATEST` constant; `env_image_tag`/`instance_image_tag` default to it and nothing else is published for Verified). - **Registry/namespace confirmed:** `docker.io/swebench/...` (default `namespace="swebench"` in `run_evaluation.py`). No alternate namespace found in the harness source for the public Verified images. - **Arch coverage confirmed:** the one instance image probed (`swebench/sweb.eval.x86_64.django_1776_django-10097:latest`) is a **single-arch `manifest.v2` (not a manifest list)**: `skopeo inspect` reports `"Architecture": "amd64"`, `"Os": "linux"`. The parallel `arm64` instance tag for the same instance does **not** exist on the registry (`skopeo inspect --raw` → access denied). **No arm64 variants exist for any repo on the public registry** for SWE-bench Verified — this is x86_64/amd64-only across the board, not a per-repo exception. -- Note the harness's Dockerfile templates (`_DOCKERFILE_BASE_PY`, etc.) **do** support building `arm64` locally (they template `--platform={platform}` and pick `conda_arch = "aarch64"` when `arch == "arm64"` — see `swebench/harness/dockerfiles/__init__.py:67-69`), so an arm64 build is *possible* in principle via local build, but nothing arm64 is ever published. **This drives Task 3 to be x86_64-only / OCP-only (or explicit QEMU emulation) for baked images sourced from the public registry.** +- Note the harness's Dockerfile templates (`_DOCKERFILE_BASE_PY`, etc.) **do** support building `arm64` locally (they template `--platform={platform}` and pick `conda_arch = "aarch64"` when `arch == "arm64"` — see `swebench/harness/dockerfiles/__init__.py:67-69`), so an arm64 build is _possible_ in principle via local build, but nothing arm64 is ever published. **This drives Task 3 to be x86_64-only / OCP-only (or explicit QEMU emulation) for baked images sourced from the public registry.** ## 3. The pinned `swebench` symbol for the env-key derivation @@ -97,7 +97,7 @@ properties (`swebench/harness/test_spec/test_spec.py`). This is **not** the raw `MAP_REPO_VERSION_TO_SPECS` constant. That constant (`swebench.harness.constants.MAP_REPO_VERSION_TO_SPECS[repo][version]`) is an -*input* to key derivation (via `docker_specs = specs.get("docker_specs", {})` +_input_ to key derivation (via `docker_specs = specs.get("docker_specs", {})` and the env/eval/repo script generators), not the key itself. ```python @@ -133,21 +133,21 @@ inst = next(r for r in ds if r["repo"] == "django/django") # picks django__dja ts = make_test_spec(inst, namespace="swebench", arch="x86_64") ``` -| Field | Value | -|---|---| -| `instance_id` | `django__django-10097` | -| `repo` | `django/django` | -| `version` | `2.2` | -| `ts.base_image_key` | `sweb.base.py.x86_64:latest` | -| `ts.env_image_key` | `sweb.env.py.x86_64.56a3bf2c8561bd901139b0:latest` | -| `ts.instance_image_key` | `swebench/sweb.eval.x86_64.django_1776_django-10097:latest` | -| `ts.is_remote_image` | `True` (because `namespace="swebench"` was passed) | -| arm64 equivalents | `sweb.env.py.arm64.56a3bf2c8561bd901139b0:latest` / `swebench/sweb.eval.arm64.django_1776_django-10097:latest` — **neither is published** (§2) | +| Field | Value | +| ----------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------- | +| `instance_id` | `django__django-10097` | +| `repo` | `django/django` | +| `version` | `2.2` | +| `ts.base_image_key` | `sweb.base.py.x86_64:latest` | +| `ts.env_image_key` | `sweb.env.py.x86_64.56a3bf2c8561bd901139b0:latest` | +| `ts.instance_image_key` | `swebench/sweb.eval.x86_64.django_1776_django-10097:latest` | +| `ts.is_remote_image` | `True` (because `namespace="swebench"` was passed) | +| arm64 equivalents | `sweb.env.py.arm64.56a3bf2c8561bd901139b0:latest` / `swebench/sweb.eval.arm64.django_1776_django-10097:latest` — **neither is published** (§2) | **Empirical stability check (why option (a) in §1 works):** sampled every instance (up to 8 per group) in several `(repo, version)` groups from Verified and computed `env_image_key` for each — every instance within a group -produced the *identical* `env_image_key`: +produced the _identical_ `env_image_key`: ``` astropy/astropy 5.0: n=4 sampled=4 distinct_env_keys=1 @@ -178,17 +178,17 @@ images are x86_64-only per §2): docker pull --platform linux/amd64 docker.io/swebench/sweb.eval.x86_64.django_1776_django-10097:latest ``` -| Fact | Value | -|---|---| -| Conda base install prefix | `/opt/miniconda3` | -| Conda binary | `/opt/miniconda3/condabin/conda`, version `23.11.0` (matches harness default `DEFAULT_DOCKER_SPECS["conda_version"] = "py311_23.11.0-2"`) | -| Conda env name inside the image | `testbed` (hardcoded as `env_name = "testbed"` in `make_test_spec`, universal across all repos/languages, not per-repo) | -| Env prefix | `/opt/miniconda3/envs/testbed` (confirmed via `python -c "import sys;print(sys.prefix)"` run inside the container) | -| Repo checkout dir | `/testbed` (`repo_directory = f"/{env_name}"`) | +| Fact | Value | +| --------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| Conda base install prefix | `/opt/miniconda3` | +| Conda binary | `/opt/miniconda3/condabin/conda`, version `23.11.0` (matches harness default `DEFAULT_DOCKER_SPECS["conda_version"] = "py311_23.11.0-2"`) | +| Conda env name inside the image | `testbed` (hardcoded as `env_name = "testbed"` in `make_test_spec`, universal across all repos/languages, not per-repo) | +| Env prefix | `/opt/miniconda3/envs/testbed` (confirmed via `python -c "import sys;print(sys.prefix)"` run inside the container) | +| Repo checkout dir | `/testbed` (`repo_directory = f"/{env_name}"`) | | Miniconda installer arch template | `x86_64` → `Miniconda3-{conda_version}-Linux-x86_64.sh`; `arm64` → `...Linux-aarch64.sh` (`swebench/harness/dockerfiles/__init__.py:67-69`) — only the x86_64 path is ever exercised by published images (§2). | **Relocation method: `conda-pack` / `conda-unpack`. Verified working -end-to-end** (packed the `testbed` env, unpacked it to a *different* prefix in +end-to-end** (packed the `testbed` env, unpacked it to a _different_ prefix in the same container, and confirmed the relocated interpreter reports the new prefix and still imports the project): @@ -229,11 +229,11 @@ is what keeps the "drift guard" in Task 2/3 meaningful. ## 5. Arch coverage summary table -| Artifact | x86_64 / amd64 | arm64 | -|---|---|---| -| Published instance images (`swebench/sweb.eval.*`) | Yes (confirmed pullable, single-arch manifest) | **No** (confirmed absent via `skopeo inspect --raw` — access denied) | -| Published env images (`sweb.env.*`) | **No** (none published at all, any arch — §1) | **No** | -| Harness's local Dockerfile *templates* | Supported | Supported in the template (`conda_arch="aarch64"`), but never exercised for published Verified images | +| Artifact | x86_64 / amd64 | arm64 | +| -------------------------------------------------- | ---------------------------------------------- | ----------------------------------------------------------------------------------------------------- | +| Published instance images (`swebench/sweb.eval.*`) | Yes (confirmed pullable, single-arch manifest) | **No** (confirmed absent via `skopeo inspect --raw` — access denied) | +| Published env images (`sweb.env.*`) | **No** (none published at all, any arch — §1) | **No** | +| Harness's local Dockerfile _templates_ | Supported | Supported in the template (`conda_arch="aarch64"`), but never exercised for published Verified images | **Consequence:** any Task 3 pipeline that sources from the public registry is inherently x86_64-only. On arm64 dev hosts (e.g. this Mac), all @@ -255,7 +255,7 @@ registry/namespace (§2), granularity verdict + Task-3 consequence (§1), exact - Granularity verdict is backed by both static source analysis (`docker_build.py`) and a live negative registry probe (`skopeo inspect - --raw` → access denied for the env-image guess and for an arm64 instance +--raw` → access denied for the env-image guess and for an arm64 instance tag), not just one or the other. - The pinned symbol (`TestSpec.env_image_key` via `make_test_spec`) was exercised against the real `SWE-bench_Verified` dataset (not a synthetic @@ -273,10 +273,10 @@ registry/namespace (§2), granularity verdict + Task-3 consequence (§1), exact - The conda relocation recipe was run to completion inside the actual published image (not simulated): `conda-pack` produced a real 103MB tarball, `conda-unpack` rewrote paths, and both `sys.prefix` and `import - django` were checked post-relocation. +django` were checked post-relocation. - No values were fabricated; the one thing NOT independently re-verified is whether `MAP_REPO_VERSION_TO_SPECS` values are byte-identical to what - built the *currently published* Docker Hub images (the registry images + built the _currently published_ Docker Hub images (the registry images could predate the installed `swebench==4.1.0`'s constants) — Task 2/3 should treat a computed `env_image_key` mismatch against a real pull as the drift signal it's designed to catch, not assume perpetual agreement. diff --git a/docs/plans/README.md b/docs/plans/README.md index 2aba5af..f1eaa53 100644 --- a/docs/plans/README.md +++ b/docs/plans/README.md @@ -6,12 +6,12 @@ README is gitignored (see the repo `.gitignore`). ## Why plans aren't committed -A plan answers *how, in what order*. The moment the code merges, the code becomes the source -of truth for *how* — the plan is now a stale, lower-fidelity copy. The durable value of our -docs is the ***why***: the decisions and rejected alternatives. Those live in: +A plan answers _how, in what order_. The moment the code merges, the code becomes the source +of truth for _how_ — the plan is now a stale, lower-fidelity copy. The durable value of our +docs is the _**why**_: the decisions and rejected alternatives. Those live in: -- **[`../specs/`](../specs/)** — dated design docs (*what & why*, in depth). -- **[`../adrs/`](../adrs/)** — permanent decision records (*one decision + consequences*). +- **[`../specs/`](../specs/)** — dated design docs (_what & why_, in depth). +- **[`../adrs/`](../adrs/)** — permanent decision records (_one decision + consequences_). So a plan's whole life is: write it → execute it → **delete it**. Committing it would just create a maintenance burden that goes stale and misleads. @@ -27,4 +27,4 @@ New plans authored via the writing-plans workflow land here by convention. --- -*Assisted-By: Claude (Anthropic AI) * +_Assisted-By: Claude (Anthropic AI) _ diff --git a/docs/specs/2026-06-16-m1-redis-session-backend-design.md b/docs/specs/2026-06-16-m1-redis-session-backend-design.md index 9f8041c..8fd7a60 100644 --- a/docs/specs/2026-06-16-m1-redis-session-backend-design.md +++ b/docs/specs/2026-06-16-m1-redis-session-backend-design.md @@ -12,7 +12,7 @@ Discovery basis: [`NOTES-pi-sessionmanager.md`](../../packages/session-backend/N Make the serverless harness's hard dependency real: Pi persists session state to **Redis instead of local JSONL**, transparently, so a session can be resumed by a -*fresh process* with no local files. +_fresh process_ with no local files. M1 is **done** when a turn round-trips through Redis and survives process death, proven by an automated deterministic test (faux provider) plus one real headless @@ -38,15 +38,15 @@ smoke run. ## 2. Key decisions (resolved during brainstorming) -| # | Decision | Choice | -|---|----------|--------| -| D1 | Entry representation | **Native passthrough** — store Pi's `FileEntry` verbatim; backend never reshapes it. | -| D2 | Fork invasiveness | **Upstreamable refactor** — introduce `SessionStorageBackend` in Pi core + `FileSessionStorageBackend` default + factory injection. | -| D3 | Sync→async bridge | **Write-behind via a harness-side buffering decorator** — keep Pi's sync append API and a **pristine** (#2032-identical) core interface; the decorator owns the queue + drain worker + `flush()`; the harness flushes at `turn_end` and `session_shutdown`. | -| D4 | Storage envelope | **Thin envelope** around the opaque Pi entry, keeping `position` + `content_sha256`. | -| D5 | Checkpoint representation | Pi **`custom` entry** with `customType: "checkpoint"` (distinct from native `compaction`). | -| D6 | M1 gate | **Deterministic integration test** (faux provider + disposable Redis) **+ one real headless smoke**. | -| D7 | Ownership split | Generic seam in `pi-fork`; Redis specifics in `@sh/session-backend` + `harness`. Dependency arrow points one way only. | +| # | Decision | Choice | +| --- | ------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| D1 | Entry representation | **Native passthrough** — store Pi's `FileEntry` verbatim; backend never reshapes it. | +| D2 | Fork invasiveness | **Upstreamable refactor** — introduce `SessionStorageBackend` in Pi core + `FileSessionStorageBackend` default + factory injection. | +| D3 | Sync→async bridge | **Write-behind via a harness-side buffering decorator** — keep Pi's sync append API and a **pristine** (#2032-identical) core interface; the decorator owns the queue + drain worker + `flush()`; the harness flushes at `turn_end` and `session_shutdown`. | +| D4 | Storage envelope | **Thin envelope** around the opaque Pi entry, keeping `position` + `content_sha256`. | +| D5 | Checkpoint representation | Pi **`custom` entry** with `customType: "checkpoint"` (distinct from native `compaction`). | +| D6 | M1 gate | **Deterministic integration test** (faux provider + disposable Redis) **+ one real headless smoke**. | +| D7 | Ownership split | Generic seam in `pi-fork`; Redis specifics in `@sh/session-backend` + `harness`. Dependency arrow points one way only. | --- @@ -93,6 +93,7 @@ The Redis backend is **injected**, never imported by Pi core. ### Behavior-preserving constraint The file backend must reproduce Pi's existing persistence semantics exactly, including: + - **Lazy flush** — entries held in memory until the first assistant message, then the whole prefix written with flag `"wx"`, `flushed = true`, append-only thereafter. - **`_rewriteFile`** (flag `"w"`) on migration and branched-session creation only — @@ -127,11 +128,11 @@ pi core: SessionStorageBackend // pristine 5 methods, exactly #2032 — no f - **Durability barriers**: the **harness** registers the `turn_end` and `session_shutdown` hooks (both confirmed Pi events, already harness-owned `pi.on(...)` listeners) and calls `bufferedBackend.flush()` from them. **Pi core never calls flush — it only emits the - events it already emits.** A *completed turn* is therefore always durable (satisfies + events it already emits.** A _completed turn_ is therefore always durable (satisfies experiment E4); only an in-flight turn can be lost on a hard kill. **Why Pi never needs to flush for its own correctness:** fork / branch / compact all -operate on the in-memory tree, which is authoritative during a live session. The *only* +operate on the in-memory tree, which is authoritative during a live session. The _only_ reason to flush is external durability (surviving process death) — a purely serverless/ harness concern. So the harness is the natural owner of both the buffer and `flush()`, and the #2032 interface contributed upstream is unchanged. @@ -143,12 +144,12 @@ envelope wrapping Pi's opaque native entry: ```ts export interface LogEntry { - position: number; // monotonic offset (Redis INCR); powers read(fromPosition) + position: number; // monotonic offset (Redis INCR); powers read(fromPosition) session_id: string; - piType: string; // denormalized copy of the Pi entry's `type`, for cheap filtering - entry: FileEntry; // Pi's native entry/header, stored & returned VERBATIM + piType: string; // denormalized copy of the Pi entry's `type`, for cheap filtering + entry: FileEntry; // Pi's native entry/header, stored & returned VERBATIM content_sha256: string; // integrity; makes E3/E4 fidelity provable by hash - timestamp: number; // wall-clock ms; audit only, not ordering + timestamp: number; // wall-clock ms; audit only, not ordering } ``` @@ -192,12 +193,12 @@ Anything Redis-flavored lives in `harness` or the backend package. This keeps th diff a clean, self-contained refactor that could be opened as a PR, and isolates the experiment's glue from the contribution. -| Lives in | What | Why | -|----------|------|-----| -| **`pi-fork` core** | `SessionStorageBackend` interface + `FileSessionStorageBackend` (extraction) + factory injection | The upstreamable slice; Pi core owns the interface and its default. | -| **`pi-fork` test dir** | Backend **contract/parity** test parametrized over `InMemory` + `File` (no Redis dependency) | Proves the extraction is behavior-preserving; ships *with* the #2032 contribution; runs in Pi's own suite. | -| **`@sh/session-backend`** | `RedisSessionBackend` (plain interface impl) + the envelope refactor (`entry.ts`) | Our code, injected — Pi never imports it. | -| **`harness` package** | `BufferedRedisBackend` decorator (queue + drain worker + `flush()`) + the wiring (inject the decorator into Pi's factory; call `flush()` from harness-owned `turn_end` / `session_shutdown` hooks; headless entry wrapper) **+ the Redis integration / mobility / recovery tests + the headless smoke** | Depends on *both* `pi-fork` and `@sh/session-backend`; keeps that dependency — and all async/buffering concerns — out of Pi core. | +| Lives in | What | Why | +| ------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------- | +| **`pi-fork` core** | `SessionStorageBackend` interface + `FileSessionStorageBackend` (extraction) + factory injection | The upstreamable slice; Pi core owns the interface and its default. | +| **`pi-fork` test dir** | Backend **contract/parity** test parametrized over `InMemory` + `File` (no Redis dependency) | Proves the extraction is behavior-preserving; ships _with_ the #2032 contribution; runs in Pi's own suite. | +| **`@sh/session-backend`** | `RedisSessionBackend` (plain interface impl) + the envelope refactor (`entry.ts`) | Our code, injected — Pi never imports it. | +| **`harness` package** | `BufferedRedisBackend` decorator (queue + drain worker + `flush()`) + the wiring (inject the decorator into Pi's factory; call `flush()` from harness-owned `turn_end` / `session_shutdown` hooks; headless entry wrapper) **+ the Redis integration / mobility / recovery tests + the headless smoke** | Depends on _both_ `pi-fork` and `@sh/session-backend`; keeps that dependency — and all async/buffering concerns — out of Pi core. | The `harness` package is already declared in `pnpm-workspace.yaml` but does not yet exist; M1 creates it. @@ -210,7 +211,7 @@ M1 creates it. Pi ships a real faux provider (`pi-fork/packages/ai/src/providers/faux.ts`: `registerFauxProvider`, `fauxAssistantMessage`, `fauxText`, `fauxToolCall`, -`FauxResponseFactory` for scripted multi-step turns *including tool calls*). The gate +`FauxResponseFactory` for scripted multi-step turns _including tool calls_). The gate rests on this shipped seam, not on a live model. 1. **Storage-swap parity** — drive a scripted turn (faux assistant message + a tool call) @@ -242,13 +243,13 @@ confirming the headless one-shot entry point drives the externalized store end-t ## 7. Risks & mitigations -| Risk | Impact | Mitigation | -|------|--------|------------| -| Fork diff diverges from upstream as Pi evolves | Maintenance burden | Keep the diff minimal + behavior-preserving; shape to #2032; pin to `406a2214`; branch (not detached HEAD). | -| Write-behind loses an in-flight turn on hard kill | Sub-turn data loss | Acceptable by design — durability boundary is the *completed turn* (E4). Flush at `turn_end` + `session_shutdown`. | -| Lazy-flush / `_rewriteFile` semantics subtly broken in extraction | Pi behavior regression | The parametrized parity test over `File` + `InMemory` is the guard; extract behavior unchanged. | +| Risk | Impact | Mitigation | +| --------------------------------------------------------------------- | --------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------ | +| Fork diff diverges from upstream as Pi evolves | Maintenance burden | Keep the diff minimal + behavior-preserving; shape to #2032; pin to `406a2214`; branch (not detached HEAD). | +| Write-behind loses an in-flight turn on hard kill | Sub-turn data loss | Acceptable by design — durability boundary is the _completed turn_ (E4). Flush at `turn_end` + `session_shutdown`. | +| Lazy-flush / `_rewriteFile` semantics subtly broken in extraction | Pi behavior regression | The parametrized parity test over `File` + `InMemory` is the guard; extract behavior unchanged. | | Fire-and-forget `append` hides a Redis write error from the call site | Silent loss before the next flush | Errors surface inside `BufferedRedisBackend` (retry/backoff); the `turn_end` flush is the durability checkpoint and can fail loudly. | -| Resume-by-`session_id` changes the factory contract | Breaks callers expecting a path | Default factory behavior (file, path-based) preserved; `session_id` resume is the injected-backend path. | +| Resume-by-`session_id` changes the factory contract | Breaks callers expecting a path | Default factory behavior (file, path-based) preserved; `session_id` resume is the injected-backend path. | --- @@ -261,4 +262,4 @@ confirming the headless one-shot entry point drives the externalized store end-t --- -*Assisted-By: Claude (Anthropic AI) * +_Assisted-By: Claude (Anthropic AI) _ diff --git a/docs/specs/2026-06-17-m2-k8s-sandbox-client-design.md b/docs/specs/2026-06-17-m2-k8s-sandbox-client-design.md index 8bd2241..2f03ec3 100644 --- a/docs/specs/2026-06-17-m2-k8s-sandbox-client-design.md +++ b/docs/specs/2026-06-17-m2-k8s-sandbox-client-design.md @@ -48,15 +48,15 @@ operations plus one real `kubectl exec` smoke on a kind cluster. ## 2. Key decisions (resolved during brainstorming) -| # | Decision | Choice | -|---|----------|--------| -| D1 | Transport to the pod | **`kubectl exec` shell-out**, behind an injectable `execInPod` seam. Zero new runtime deps; a near-verbatim port of the SSH example's `sshExec`. | -| D2 | Operation coverage | **Route all seven** (`read/write/edit/bash/ls/grep/find`). The SSH example leaves `ls/grep/find` local; for a real sandbox that silently shows the head's FS, so M2 closes the gap. | -| D3 | Pod lifecycle | **Client targets an existing pod** identified by env (`namespace` + `pod`); ship one plain `Deployment + PVC` manifest as the fixture, applied out-of-band. No dynamic create/teardown in the client. | -| D4 | Verification gate | **Injectable seam + deterministic fake-exec unit tests (all 7 ops) + one real kind smoke.** Direct mirror of M1's gate. | -| D5 | Code placement | **New `@sh/k8s-sandbox` package + thin env-gated `cli.ts` wiring.** **No pi-fork change** — the Operations seam is already native. | -| D6 | Headless config surface | **Env vars / factory argument, not CLI flags.** `KAGENTI_SANDBOX_POD` is the on/off gate; unset ⇒ extension inert, all tools local. | -| D7 | Path mapping | **Mirror the SSH example:** announce the pod cwd in the system prompt via `before_agent_start`; map head-cwd → pod-cwd in the operations (naive `path.replace`, fragility accepted for M2). | +| # | Decision | Choice | +| --- | ----------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| D1 | Transport to the pod | **`kubectl exec` shell-out**, behind an injectable `execInPod` seam. Zero new runtime deps; a near-verbatim port of the SSH example's `sshExec`. | +| D2 | Operation coverage | **Route all seven** (`read/write/edit/bash/ls/grep/find`). The SSH example leaves `ls/grep/find` local; for a real sandbox that silently shows the head's FS, so M2 closes the gap. | +| D3 | Pod lifecycle | **Client targets an existing pod** identified by env (`namespace` + `pod`); ship one plain `Deployment + PVC` manifest as the fixture, applied out-of-band. No dynamic create/teardown in the client. | +| D4 | Verification gate | **Injectable seam + deterministic fake-exec unit tests (all 7 ops) + one real kind smoke.** Direct mirror of M1's gate. | +| D5 | Code placement | **New `@sh/k8s-sandbox` package + thin env-gated `cli.ts` wiring.** **No pi-fork change** — the Operations seam is already native. | +| D6 | Headless config surface | **Env vars / factory argument, not CLI flags.** `KAGENTI_SANDBOX_POD` is the on/off gate; unset ⇒ extension inert, all tools local. | +| D7 | Path mapping | **Mirror the SSH example:** announce the pod cwd in the system prompt via `before_agent_start`; map head-cwd → pod-cwd in the operations (naive `path.replace`, fragility accepted for M2). | --- @@ -109,12 +109,12 @@ canned `stdout` / `exitCode`, so no test touches a cluster. Resolved at construction from the factory argument or env (env wins for headless runs): -| Env var | Meaning | -|---------|---------| -| `KAGENTI_SANDBOX_POD` | Pod name — **the on/off gate.** Unset ⇒ extension inert; all tools run locally (default headless behavior unchanged). | -| `KAGENTI_SANDBOX_NAMESPACE` | Namespace (default `default`). | -| `KAGENTI_SANDBOX_CONTEXT` | Kube context (optional; omit to use current-context). | -| `KAGENTI_SANDBOX_CWD` | Pod working directory, e.g. `/workspace`. The cwd announced to the model; head-cwd → this is the path map. | +| Env var | Meaning | +| --------------------------- | --------------------------------------------------------------------------------------------------------------------- | +| `KAGENTI_SANDBOX_POD` | Pod name — **the on/off gate.** Unset ⇒ extension inert; all tools run locally (default headless behavior unchanged). | +| `KAGENTI_SANDBOX_NAMESPACE` | Namespace (default `default`). | +| `KAGENTI_SANDBOX_CONTEXT` | Kube context (optional; omit to use current-context). | +| `KAGENTI_SANDBOX_CWD` | Pod working directory, e.g. `/workspace`. The cwd announced to the model; head-cwd → this is the path map. | --- @@ -123,22 +123,22 @@ Resolved at construction from the factory argument or env (env wins for headless File transfer mirrors the SSH example's `cat` / `base64` approach. All paths are mapped head-cwd → pod-cwd before the command is built, and shell-quoted. -| Op | In-pod implementation | -|----|----------------------| -| **Read** | `readFile` → `cat

` (buffered); `access` → `test -r

`; `detectImageMimeType` → `file --mime-type -b

` (whitelist jpeg/png/gif/webp, else null) | -| **Write** | `writeFile` → `echo \| base64 -d >

`; `mkdir` → `mkdir -p ` | -| **Edit** | compose Read + Write; `access` → `test -r

&& test -w

` | -| **Bash** | stream `cd && ` through `execInPod` with `onData` / `signal` / `timeout` (mirrors `createRemoteBashOps`) | -| **Ls** | `exists` → `test -e

`; `stat.isDirectory()` → `test -d

`; `readdir` → `ls -1A

` split on newline | -| **Grep** | `isDirectory` → `test -d

`; `readFile` (string) → `cat

` (context lines) | -| **Find** | `exists` → `test -e

`; `glob` → run `fd` / `find` **in-pod** (Pi's default fd path is local; supplying a custom `glob` is the documented override point) | +| Op | In-pod implementation | +| --------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------- | +| **Read** | `readFile` → `cat

` (buffered); `access` → `test -r

`; `detectImageMimeType` → `file --mime-type -b

` (whitelist jpeg/png/gif/webp, else null) | +| **Write** | `writeFile` → `echo \| base64 -d >

`; `mkdir` → `mkdir -p ` | +| **Edit** | compose Read + Write; `access` → `test -r

&& test -w

` | +| **Bash** | stream `cd && ` through `execInPod` with `onData` / `signal` / `timeout` (mirrors `createRemoteBashOps`) | +| **Ls** | `exists` → `test -e

`; `stat.isDirectory()` → `test -d

`; `readdir` → `ls -1A

` split on newline | +| **Grep** | `isDirectory` → `test -d

`; `readFile` (string) → `cat

` (context lines) | +| **Find** | `exists` → `test -e

`; `glob` → run `fd` / `find` **in-pod** (Pi's default fd path is local; supplying a custom `glob` is the documented override point) | ### 4.1 Load-bearing risk — grep/find search routing (proven first, not assumed) For `read/write/edit/bash/ls` the search/exec clearly flows through `operations`. -For **grep** and **find**, Pi's tool may run `rg` / `fd` *locally in `execute()`* and -use `operations` only for ancillary reads — `find.ts` comments: *"Actual fd execution -happens in execute() when no custom glob is provided."* +For **grep** and **find**, Pi's tool may run `rg` / `fd` _locally in `execute()`_ and +use `operations` only for ancillary reads — `find.ts` comments: _"Actual fd execution +happens in execute() when no custom glob is provided."_ The implementation plan's **first task is a spike** that confirms whether supplying a custom `glob` (find) and the grep operations actually reroutes the **search itself** to @@ -193,14 +193,14 @@ No dynamic create/teardown in the client — the deferred sandbox track owns lif ## 8. Residual risks -| Risk | Mitigation | -|------|------------| -| grep/find search hardwired to a local binary | **Task 1 spike** proves routing; fallback = override the tool's `execute` to run the search in-pod. | -| Per-op `kubectl exec` latency (each op = a new exec) | Acceptable for M2 (not the perf milestone); flagged for M3 — a persistent in-pod channel or API-exec is the upgrade path. | -| Abort/timeout kills the local `kubectl`, but the remote command may linger (no TTY) | Accepted for M2; noted for the lifecycle track. Kill the local process on `signal`/timeout as the SSH example does. | -| `base64 -d` vs `--decode` (busybox) and binary-file safety | Mirror the SSH example's base64 flow; the smoke includes a binary-file round-trip. | -| Naive `path.replace` cwd mapping (inherited from SSH) | Kept for M2; fragility documented. Revisit if paths outside the announced cwd appear. | -| `before_agent_start` may not fire on the headless path | Spike confirms; if absent, announce the pod cwd by other means (status line + system-prompt edit at session construction). | +| Risk | Mitigation | +| ----------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------- | +| grep/find search hardwired to a local binary | **Task 1 spike** proves routing; fallback = override the tool's `execute` to run the search in-pod. | +| Per-op `kubectl exec` latency (each op = a new exec) | Acceptable for M2 (not the perf milestone); flagged for M3 — a persistent in-pod channel or API-exec is the upgrade path. | +| Abort/timeout kills the local `kubectl`, but the remote command may linger (no TTY) | Accepted for M2; noted for the lifecycle track. Kill the local process on `signal`/timeout as the SSH example does. | +| `base64 -d` vs `--decode` (busybox) and binary-file safety | Mirror the SSH example's base64 flow; the smoke includes a binary-file round-trip. | +| Naive `path.replace` cwd mapping (inherited from SSH) | Kept for M2; fragility documented. Revisit if paths outside the announced cwd appear. | +| `before_agent_start` may not fire on the headless path | Spike confirms; if absent, announce the pod cwd by other means (status line + system-prompt edit at session construction). | --- @@ -214,4 +214,4 @@ expressible; M3 adds the Knative trigger to reach Config C (serverless). --- -*Assisted-By: Claude (Anthropic AI) * +_Assisted-By: Claude (Anthropic AI) _ diff --git a/docs/specs/2026-06-17-m3-persistent-channel-design.md b/docs/specs/2026-06-17-m3-persistent-channel-design.md index f150785..0193f55 100644 --- a/docs/specs/2026-06-17-m3-persistent-channel-design.md +++ b/docs/specs/2026-06-17-m3-persistent-channel-design.md @@ -26,7 +26,7 @@ M3 closes all three, **harness-side only** (`pi-fork` untouched, as in M2): inside the pod. 3. **find ignore-list** — find honours the `ignore` patterns (negated globs) AND `.gitignore` for ignored **directories** (verified on the pod's ripgrep 14.1.0). Minor divergence from - Pi's `fd`: an individually-gitignored *file* matching the positive `-g ` is + Pi's `fd`: an individually-gitignored _file_ matching the positive `-g ` is re-included by the glob whitelist — see D5 caveat below. M3 is **done** when: a burst of fast ops in one agent turn is served by a single reused @@ -60,16 +60,16 @@ backed by cluster-free unit tests plus one real kind smoke. ## 2. Key decisions (resolved during brainstorming) -| # | Decision | Choice | -|---|----------|--------| -| D1 | Persistent transport | **T1 — long-lived `kubectl exec -i … -- bash`** with a framed stdin/stdout protocol. Zero new deps/infra; same kubectl shell-out and same injectable `ExecInPod` seam as M2. (Rejected: T2 pod-side RPC server + port-forward — breaks the plain-Deployment posture; T3 SPDY via `@kubernetes/client-node` — heavy dep, same framing problem.) | -| D2 | Channel scope | **S-fast — persistent channel for the small request/response ops only** (read, write, edit, ls, stat, mkdir, find, mime). **bash, grep, and `user_bash` keep M2's per-call `kubectl exec`** — they stream via `onData`, honour `signal` abort, run long, and already amortize the exec overhead. Captures the bulk of the latency win while the framing protocol only ever handles the simple case. | -| D3 | Wire framing | **Sentinel-bracketed, base64-encoded payload + trailing exit code.** base64's alphabet cannot contain the marker bytes ⇒ collision-free framing and binary-safe stdout. One command in flight at a time (a queue), since the ops are synchronous request/response. | -| D4 | Env injection site | **Bash-ops layer, as an `env VAR=val … bash -c ` prefix** — transport-agnostic (works for both kubectl and persistent transports) and non-leaking (scoped to the one invocation). **No `ExecInPod` signature change.** | -| D5 | find implementation | **`rg --files -g -g '!'`** — reuses the ripgrep already in the image, honours `.gitignore`, and applies the `ignore` negations. **Verified behavior on rg 14.1.0 (nuanced):** gitignored *directories* (e.g. `node_modules/`, `dist/`) are pruned and stay excluded even when `-g` matches files inside them; but an individually-gitignored *file* matching the positive `-g ` IS re-included (the glob whitelist-overrides a file-level ignore) — a minor divergence from Pi's `fd --glob`. The `ignore` list (negated `-g '!'`) always excludes its entries. Accepted for M3; closing the individual-file edge is a low-priority follow-up. (Rejected: keep `find` + translate globs to `-path -prune` — brittle.) | -| D6 | Resilience | **Transparent fallback.** A failed spawn or a session that dies mid-command degrades that op to a one-shot `kubectl exec` (`kubectlExecInPod`), and the session re-spawns lazily. A dead channel never hard-fails — it reverts to M2 behavior. | -| D7 | Lifecycle | **Dispose on `session_shutdown`** (fires on quit/reload/new/resume/fork) — end stdin and kill the kubectl process so it is never leaked. Spawn is **lazy** (first fast op), so an inert run spawns nothing. | -| D8 | Verification gate | **Pure framing unit tests + fake-`spawn` persistent-exec tests + operations tests + one real kind smoke.** Direct mirror of M2's D4 — cluster-free and build-free except the single smoke. | +| # | Decision | Choice | +| --- | -------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| D1 | Persistent transport | **T1 — long-lived `kubectl exec -i … -- bash`** with a framed stdin/stdout protocol. Zero new deps/infra; same kubectl shell-out and same injectable `ExecInPod` seam as M2. (Rejected: T2 pod-side RPC server + port-forward — breaks the plain-Deployment posture; T3 SPDY via `@kubernetes/client-node` — heavy dep, same framing problem.) | +| D2 | Channel scope | **S-fast — persistent channel for the small request/response ops only** (read, write, edit, ls, stat, mkdir, find, mime). **bash, grep, and `user_bash` keep M2's per-call `kubectl exec`** — they stream via `onData`, honour `signal` abort, run long, and already amortize the exec overhead. Captures the bulk of the latency win while the framing protocol only ever handles the simple case. | +| D3 | Wire framing | **Sentinel-bracketed, base64-encoded payload + trailing exit code.** base64's alphabet cannot contain the marker bytes ⇒ collision-free framing and binary-safe stdout. One command in flight at a time (a queue), since the ops are synchronous request/response. | +| D4 | Env injection site | **Bash-ops layer, as an `env VAR=val … bash -c ` prefix** — transport-agnostic (works for both kubectl and persistent transports) and non-leaking (scoped to the one invocation). **No `ExecInPod` signature change.** | +| D5 | find implementation | **`rg --files -g -g '!'`** — reuses the ripgrep already in the image, honours `.gitignore`, and applies the `ignore` negations. **Verified behavior on rg 14.1.0 (nuanced):** gitignored _directories_ (e.g. `node_modules/`, `dist/`) are pruned and stay excluded even when `-g` matches files inside them; but an individually-gitignored _file_ matching the positive `-g ` IS re-included (the glob whitelist-overrides a file-level ignore) — a minor divergence from Pi's `fd --glob`. The `ignore` list (negated `-g '!'`) always excludes its entries. Accepted for M3; closing the individual-file edge is a low-priority follow-up. (Rejected: keep `find` + translate globs to `-path -prune` — brittle.) | +| D6 | Resilience | **Transparent fallback.** A failed spawn or a session that dies mid-command degrades that op to a one-shot `kubectl exec` (`kubectlExecInPod`), and the session re-spawns lazily. A dead channel never hard-fails — it reverts to M2 behavior. | +| D7 | Lifecycle | **Dispose on `session_shutdown`** (fires on quit/reload/new/resume/fork) — end stdin and kill the kubectl process so it is never leaked. Spawn is **lazy** (first fast op), so an inert run spawns nothing. | +| D8 | Verification gate | **Pure framing unit tests + fake-`spawn` persistent-exec tests + operations tests + one real kind smoke.** Direct mirror of M2's D4 — cluster-free and build-free except the single smoke. | --- @@ -129,7 +129,7 @@ cluster** — so the gnarly parsing is unit-tested in isolation. ```ts export function persistentExecInPod( config: K8sSandboxConfig, - deps: { fallback: ExecInPod; spawn?: typeof import("node:child_process").spawn }, + deps: { fallback: ExecInPod; spawn?: typeof import('node:child_process').spawn }, ): ExecInPod & { dispose: () => void }; ``` @@ -146,7 +146,7 @@ transport it gets. Internals: paths with no `cd`), exactly as M2's `kubectlExecInPod` runs `bash -c ` verbatim. - **Per-command stdin via heredoc.** The `ExecInPod` contract carries `opts.stdin` (used only by `writeFile`'s `base64 -d > `). A per-call `kubectl exec -i` feeds that as the - command's stdin, but a shared session's stdin *is* the command stream. So when `opts.stdin` + command's stdin, but a shared session's stdin _is_ the command stream. So when `opts.stdin` is set, the transport appends a nonce-delimited heredoc to the command — ` <<''\n\n` — and bash reads the heredoc body from the same stream and feeds it to the command. Binary-safe (writeFile's payload is already base64) and ARG_MAX-safe (data is not an argv). Contract: when `stdin` is provided, `command` must be a single pipeline whose @@ -160,7 +160,7 @@ transport it gets. Internals: Crucially, the persistent transport is wrapped so that **any rejection caused by channel unavailability is retried once via `deps.fallback`** (D6) — the file op still succeeds via -a one-shot `kubectl exec`. Genuine command failures (non-zero exit codes) are *not* retried; +a one-shot `kubectl exec`. Genuine command failures (non-zero exit codes) are _not_ retried; they pass through unchanged. ### 3.3 No new config @@ -179,24 +179,29 @@ nothing for an operator to tune or disable.) M2 built one `exec` and gave it to all seven tools. M3 builds two and routes by op shape: ```ts -const streamExec = opts?.exec ?? kubectlExecInPod(config); // M2 path (per-call) -const fastExec = opts?.exec ?? persistentExecInPod(config, { // M3 path (persistent) - fallback: kubectlExecInPod(config), -}); +const streamExec = opts?.exec ?? kubectlExecInPod(config); // M2 path (per-call) +const fastExec = + opts?.exec ?? + persistentExecInPod(config, { + // M3 path (persistent) + fallback: kubectlExecInPod(config), + }); // fast request/response ops → persistent channel -registerTool(createReadTool (localCwd, { operations: createPodReadOps (fastExec, config) })); +registerTool(createReadTool(localCwd, { operations: createPodReadOps(fastExec, config) })); registerTool(createWriteTool(localCwd, { operations: createPodWriteOps(fastExec, config) })); -registerTool(createEditTool (localCwd, { operations: createPodEditOps (fastExec, config) })); -registerTool(createLsTool (localCwd, { operations: createPodLsOps (fastExec, config) })); -registerTool(createFindTool (localCwd, { operations: createPodFindOps (fastExec, config) })); +registerTool(createEditTool(localCwd, { operations: createPodEditOps(fastExec, config) })); +registerTool(createLsTool(localCwd, { operations: createPodLsOps(fastExec, config) })); +registerTool(createFindTool(localCwd, { operations: createPodFindOps(fastExec, config) })); // streaming / long-running ops → per-call kubectl exec (unchanged from M2) -registerTool(createBashTool (localCwd, { operations: createPodBashOps (streamExec, config) })); +registerTool(createBashTool(localCwd, { operations: createPodBashOps(streamExec, config) })); registerTool(createPodGrepTool(localCwd, streamExec, config)); -pi.on("user_bash", () => ({ operations: createPodBashOps(streamExec, config) })); +pi.on('user_bash', () => ({ operations: createPodBashOps(streamExec, config) })); -pi.on("session_shutdown", () => { if ("dispose" in fastExec) fastExec.dispose(); }); +pi.on('session_shutdown', () => { + if ('dispose' in fastExec) fastExec.dispose(); +}); ``` When `opts.exec` is supplied (tests / alternate auth), it is used for **both** tiers, so @@ -211,13 +216,16 @@ Pi passes `env` only to the bash tool. When present, prefix a non-leaking, per-i ```ts exec: async (command, cwd, { onData, signal, timeout, env }) => { const prefix = env - ? "env " + Object.entries(env) + ? 'env ' + + Object.entries(env) .filter(([, v]) => v !== undefined) - .map(([k, v]) => `${k}=${shQuote(String(v))}`).join(" ") + " " - : ""; + .map(([k, v]) => `${k}=${shQuote(String(v))}`) + .join(' ') + + ' ' + : ''; const wrapped = prefix ? `cd ${q(cwd)} && ${prefix}bash -c ${shQuote(command)}` - : `cd ${q(cwd)} && ${command}`; // M2's exact form when env is absent (no behavior change) + : `cd ${q(cwd)} && ${command}`; // M2's exact form when env is absent (no behavior change) const r = await exec(wrapped, { onData, signal, timeout }); return { exitCode: r.exitCode }; }; @@ -233,16 +241,20 @@ Replace `find . -type f -name ` with `rg --files`: ```ts glob: async (pattern, cwd, { ignore, limit }) => { - const globs = [`-g ${shQuote(pattern)}`, ...ignore.map((ig) => `-g ${shQuote("!" + ig)}`)]; - const r = await exec(`cd ${q(cwd)} && rg --files --hidden ${globs.join(" ")} | head -n ${limit}`); - return r.stdout.toString().split("\n").filter((x) => x.length > 0).map((rel) => rel.replace(/^\.\//, "")); + const globs = [`-g ${shQuote(pattern)}`, ...ignore.map((ig) => `-g ${shQuote('!' + ig)}`)]; + const r = await exec(`cd ${q(cwd)} && rg --files --hidden ${globs.join(' ')} | head -n ${limit}`); + return r.stdout + .toString() + .split('\n') + .filter((x) => x.length > 0) + .map((rel) => rel.replace(/^\.\//, '')); }; ``` `rg --files` lists files under cwd honouring `.gitignore`; `--hidden` keeps dotfiles in view; each `ignore` pattern becomes a negated glob. **Verified nuance (rg 14.1.0):** -gitignored *directories* (e.g. `node_modules/`, `dist/`) are pruned and stay excluded even -when `-g` matches files inside; but an individually-gitignored *file* matching the positive +gitignored _directories_ (e.g. `node_modules/`, `dist/`) are pruned and stay excluded even +when `-g` matches files inside; but an individually-gitignored _file_ matching the positive `-g ` is re-included (the glob whitelist-overrides a file-level ignore) — a minor divergence from Pi's `fd --glob`. The `ignore`-list negated globs (`-g '!'`) always exclude their entries. Output shape (relative paths, `./` stripped, `limit`-capped) matches @@ -252,14 +264,14 @@ M2. `exists` is unchanged. ## 5. Error handling & resilience -| Condition | Behavior | -|-----------|----------| +| Condition | Behavior | +| ------------------------------------------------ | ----------------------------------------------------------------------------------------------------- | | Persistent spawn fails (no kubectl, bad context) | Op transparently retried via `fallback` (one-shot exec); session re-spawn attempted lazily next call. | -| Session dies mid-command (pod restart, net blip) | In-flight command rejects → retried once via `fallback`; session marked dead and re-spawned lazily. | -| `opts.timeout` on a fast op | Kill + re-spawn session; reject `timeout:` (M2-compatible). | -| `opts.signal` aborted | Kill + re-spawn session; reject `aborted` (M2-compatible). | -| Genuine non-zero exit code | Passed through unchanged (**not** treated as channel failure; no retry). | -| `session_shutdown` | `dispose()` — end stdin, kill child; no leaked process. | +| Session dies mid-command (pod restart, net blip) | In-flight command rejects → retried once via `fallback`; session marked dead and re-spawned lazily. | +| `opts.timeout` on a fast op | Kill + re-spawn session; reject `timeout:` (M2-compatible). | +| `opts.signal` aborted | Kill + re-spawn session; reject `aborted` (M2-compatible). | +| Genuine non-zero exit code | Passed through unchanged (**not** treated as channel failure; no retry). | +| `session_shutdown` | `dispose()` — end stdin, kill child; no leaked process. | Streaming ops (bash/grep) retain M2's exact error handling because their transport is unchanged. @@ -289,22 +301,22 @@ unchanged. ## 7. Residual risks -| Risk | Mitigation | -|------|------------| -| Framing parser mis-handles an edge (huge output, marker-like bytes) | base64 makes payloads marker-free by construction; `FrameParser` is pure and exhaustively unit-tested incl. chunk-boundary splits and large/binary payloads. | -| Shared bash accumulates state across commands (cwd drift, leaked vars) | Each fast op is fully self-contained (`cd && …`); env is injected per-command via `env …`, never `export`. No op relies on prior-command state. | -| Persistent session masks a real connectivity problem by always falling back | Fallback is logged; the smoke asserts the channel is actually used (single process), so a silent permanent-fallback regression is caught. | -| `rg --files` semantics differ subtly from `fd`/`find` (e.g. symlinks, no-match exit) | Output shape pinned by unit tests; no-match returns empty list (rg `--files` exits 0 with no output); smoke verifies gitignore + ignore-list behavior end-to-end. | -| `head -n ` closing the pipe early sends SIGPIPE to `rg` | Acceptable — rg's partial output is the intended truncation; exit status of the op is the wrapper's `$?` of the pipeline's last stage, which M2 already tolerates for capped output. | -| Per-call streaming bash still pays per-op exec cost | Accepted (D2): bash amortizes exec over real work; S-all (stream over the channel) is a deferred follow-up if profiling shows bash exec overhead dominates. | +| Risk | Mitigation | +| ------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | +| Framing parser mis-handles an edge (huge output, marker-like bytes) | base64 makes payloads marker-free by construction; `FrameParser` is pure and exhaustively unit-tested incl. chunk-boundary splits and large/binary payloads. | +| Shared bash accumulates state across commands (cwd drift, leaked vars) | Each fast op is fully self-contained (`cd && …`); env is injected per-command via `env …`, never `export`. No op relies on prior-command state. | +| Persistent session masks a real connectivity problem by always falling back | Fallback is logged; the smoke asserts the channel is actually used (single process), so a silent permanent-fallback regression is caught. | +| `rg --files` semantics differ subtly from `fd`/`find` (e.g. symlinks, no-match exit) | Output shape pinned by unit tests; no-match returns empty list (rg `--files` exits 0 with no output); smoke verifies gitignore + ignore-list behavior end-to-end. | +| `head -n ` closing the pipe early sends SIGPIPE to `rg` | Acceptable — rg's partial output is the intended truncation; exit status of the op is the wrapper's `$?` of the pipeline's last stage, which M2 already tolerates for capped output. | +| Per-call streaming bash still pays per-op exec cost | Accepted (D2): bash amortizes exec over real work; S-all (stream over the channel) is a deferred follow-up if profiling shows bash exec overhead dominates. | --- ## 8. Relationship to the parent plan -The parent plan flagged exactly this work in M2's residual-risk table (§8): *"Per-op +The parent plan flagged exactly this work in M2's residual-risk table (§8): _"Per-op `kubectl exec` latency … flagged for M3 — a persistent in-pod channel or API-exec is the -upgrade path."* M3 takes the **persistent in-pod channel** branch (T1), not API-exec, +upgrade path."_ M3 takes the **persistent in-pod channel** branch (T1), not API-exec, because it preserves M2's whole posture (kubectl shell-out, no pod server, no new deps). This is a **sandbox-track hardening increment** that makes Config B (persistent decoupled: @@ -314,4 +326,4 @@ session layers and adds no new external surface. --- -*Assisted-By: Claude (Anthropic AI) * +_Assisted-By: Claude (Anthropic AI) _ diff --git a/docs/specs/2026-06-17-m4-knative-serverless-wrapper-design.md b/docs/specs/2026-06-17-m4-knative-serverless-wrapper-design.md index fce3d16..8c6f2bc 100644 --- a/docs/specs/2026-06-17-m4-knative-serverless-wrapper-design.md +++ b/docs/specs/2026-06-17-m4-knative-serverless-wrapper-design.md @@ -54,16 +54,16 @@ M4 is **done** when: ## 2. Key decisions -| # | Decision | Choice | -|---|----------|--------| -| D1 | Response model | **Simple request/response.** `POST /turn` blocks until the LLM turn completes, returns the full assistant message. No streaming, no fire-and-forget. Simplest proof of the serverless lifecycle. | -| D2 | Sandbox model | **Pre-provisioned.** The sandbox pod already exists; the Knative service receives `KAGENTI_SANDBOX_POD` as an env var. Sandbox lifecycle is a separate concern. | -| D3 | Deployment target | **Kind cluster + Knative Serving** (Kourier networking). Proves real scale-to-zero locally. | -| D4 | Container build | **Multi-stage Dockerfile.** Node 20 alpine, pnpm workspace install, pi-fork build chain, kubectl for sandbox exec. | -| D5 | Package structure | **New `@sh/knative-server`** package for the HTTP server. Shared `runTurn()` extracted to `harness/src/run-turn.ts`. `cli.ts` becomes a thin wrapper. | -| D6 | HTTP framework | **Node built-in `http` module.** Zero additional deps, keeps the image small. | -| D7 | Concurrency | **`containerConcurrency: 1`** on the Knative Service. One request per pod; Knative scales horizontally for concurrent sessions. Avoids shared-state complexity. | -| D8 | Session lifecycle | **Single endpoint.** Omit `sessionId` to create a new session; include it to resume. Response always includes `sessionId` for the caller to capture. | +| # | Decision | Choice | +| --- | ----------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | +| D1 | Response model | **Simple request/response.** `POST /turn` blocks until the LLM turn completes, returns the full assistant message. No streaming, no fire-and-forget. Simplest proof of the serverless lifecycle. | +| D2 | Sandbox model | **Pre-provisioned.** The sandbox pod already exists; the Knative service receives `KAGENTI_SANDBOX_POD` as an env var. Sandbox lifecycle is a separate concern. | +| D3 | Deployment target | **Kind cluster + Knative Serving** (Kourier networking). Proves real scale-to-zero locally. | +| D4 | Container build | **Multi-stage Dockerfile.** Node 20 alpine, pnpm workspace install, pi-fork build chain, kubectl for sandbox exec. | +| D5 | Package structure | **New `@sh/knative-server`** package for the HTTP server. Shared `runTurn()` extracted to `harness/src/run-turn.ts`. `cli.ts` becomes a thin wrapper. | +| D6 | HTTP framework | **Node built-in `http` module.** Zero additional deps, keeps the image small. | +| D7 | Concurrency | **`containerConcurrency: 1`** on the Knative Service. One request per pod; Knative scales horizontally for concurrent sessions. Avoids shared-state complexity. | +| D8 | Session lifecycle | **Single endpoint.** Omit `sessionId` to create a new session; include it to resume. Response always includes `sessionId` for the caller to capture. | --- @@ -137,17 +137,17 @@ Extracted from `cli.ts`, this is the reusable core: ```ts export interface TurnConfig { - redisUrl?: string; // default: "redis://localhost:6379" - cwd?: string; // default: process.cwd() - anthropicBaseUrl?: string; // gateway bridge (optional) - anthropicAuthToken?: string; // gateway bridge (optional) + redisUrl?: string; // default: "redis://localhost:6379" + cwd?: string; // default: process.cwd() + anthropicBaseUrl?: string; // gateway bridge (optional) + anthropicAuthToken?: string; // gateway bridge (optional) } export interface TurnResult { sessionId: string; - response: string; // assistant's final text content - stopReason: string; // "end_turn" | "error" | "aborted" | ... - errorMessage?: string; // present when stopReason is "error" + response: string; // assistant's final text content + stopReason: string; // "end_turn" | "error" | "aborted" | ... + errorMessage?: string; // present when stopReason is "error" } export async function runTurn( @@ -187,23 +187,23 @@ console.log(`SESSION_ID=${result.sessionId}`); Minimal Node `http` — no framework: ```ts -import { createServer } from "node:http"; -import { runTurn } from "@sh/harness/run-turn"; +import { createServer } from 'node:http'; +import { runTurn } from '@sh/harness/run-turn'; -const PORT = parseInt(process.env.PORT || "8080", 10); +const PORT = parseInt(process.env.PORT || '8080', 10); const server = createServer(async (req, res) => { - if (req.method === "GET" && req.url === "/health") { - res.writeHead(200).end("ok"); + if (req.method === 'GET' && req.url === '/health') { + res.writeHead(200).end('ok'); return; } - if (req.method === "POST" && req.url === "/turn") { + if (req.method === 'POST' && req.url === '/turn') { const body = await readBody(req); const { sessionId, prompt } = JSON.parse(body); if (!prompt) { - res.writeHead(400, headers).end(JSON.stringify({ error: "prompt_required" })); + res.writeHead(400, headers).end(JSON.stringify({ error: 'prompt_required' })); return; } @@ -216,11 +216,13 @@ const server = createServer(async (req, res) => { res.writeHead(200, headers).end(JSON.stringify(result)); } catch (err) { const message = err instanceof Error ? err.message : String(err); - const status = message.includes("not_found") ? 404 : 500; - res.writeHead(status, headers).end(JSON.stringify({ - error: message, - ...(sessionId ? { sessionId } : {}), - })); + const status = message.includes('not_found') ? 404 : 500; + res.writeHead(status, headers).end( + JSON.stringify({ + error: message, + ...(sessionId ? { sessionId } : {}), + }), + ); } return; } @@ -229,7 +231,7 @@ const server = createServer(async (req, res) => { }); // Graceful shutdown (Knative sends SIGTERM before killing) -process.on("SIGTERM", () => { +process.on('SIGTERM', () => { server.close(() => process.exit(0)); }); @@ -267,6 +269,7 @@ CMD ["node", "--import", "tsx", "packages/knative-server/src/server.ts"] ``` Notes: + - `kubectl` in the runtime image for `@sh/k8s-sandbox` pod exec. - Source-only packages (`harness/`, `packages/`) run via `tsx` (no compile step needed). - Pi-fork is pre-built in stage 1 (ships compiled JS). @@ -288,9 +291,9 @@ spec: template: metadata: annotations: - autoscaling.knative.dev/min-scale: "0" - autoscaling.knative.dev/max-scale: "5" - autoscaling.knative.dev/scale-to-zero-pod-retention-period: "30s" + autoscaling.knative.dev/min-scale: '0' + autoscaling.knative.dev/max-scale: '5' + autoscaling.knative.dev/scale-to-zero-pod-retention-period: '30s' spec: containerConcurrency: 1 timeoutSeconds: 300 @@ -300,9 +303,9 @@ spec: - containerPort: 8080 env: - name: REDIS_URL - value: "redis://redis.default.svc:6379" + value: 'redis://redis.default.svc:6379' - name: KAGENTI_SANDBOX_POD - value: "sandbox-0" + value: 'sandbox-0' - name: ANTHROPIC_API_KEY valueFrom: secretKeyRef: @@ -314,11 +317,11 @@ spec: port: 8080 resources: requests: - memory: "256Mi" - cpu: "100m" + memory: '256Mi' + cpu: '100m' limits: - memory: "512Mi" - cpu: "500m" + memory: '512Mi' + cpu: '500m' ``` ### `deploy/knative/setup-kind.sh` @@ -366,12 +369,12 @@ Client Knative Redis Error responses: -| Condition | HTTP status | Body | -|-----------|-------------|------| -| Missing `prompt` | 400 | `{ "error": "prompt_required" }` | -| `sessionId` not found in Redis | 404 | `{ "error": "session_not_found" }` | -| LLM / turn failure | 500 | `{ "error": "", "sessionId": "" }` | -| Invalid JSON body | 400 | `{ "error": "invalid_json" }` | +| Condition | HTTP status | Body | +| ------------------------------ | ----------- | ----------------------------------------------- | +| Missing `prompt` | 400 | `{ "error": "prompt_required" }` | +| `sessionId` not found in Redis | 404 | `{ "error": "session_not_found" }` | +| LLM / turn failure | 500 | `{ "error": "", "sessionId": "" }` | +| Invalid JSON body | 400 | `{ "error": "invalid_json" }` | --- @@ -410,6 +413,7 @@ Proves the full serverless lifecycle on Kind: 5. **Sandbox check** (if KAGENTI_SANDBOX_POD deployed) — curl `POST /turn { sessionId, prompt: "List files in /tmp" }` → assert tool execution happened (response references file listing). **Success criteria:** + - Steps 1–4 all pass (the serverless thesis holds). - All existing test suites remain green (harness, session-backend, k8s-sandbox). @@ -417,14 +421,14 @@ Proves the full serverless lifecycle on Kind: ## 10. Residual risks -| Risk | Impact | Mitigation | -|------|--------|------------| -| Cold-start latency too high (Knative + Node startup + Redis read + LLM call) | Poor interactive UX | M4 measures but does not optimize — compaction-checkpoint (M5) addresses this. Acceptable for PoC. | -| Knative idle timeout too aggressive (kills pod mid-LLM-call) | Dropped request | `timeoutSeconds: 300` on the Knative revision gives LLM calls up to 5 minutes. | -| kubectl in-container cannot reach sandbox pod (RBAC) | Sandbox tools fail | Setup script creates a ServiceAccount with pod exec permissions; fallback: run without sandbox (tools are inert without env var). | -| Pi-fork build fragility (build order, version drift) | Dockerfile breaks | Pinned pi-fork commit (submodule); explicit ordered build; CI will catch drift. | -| Redis not reachable from Knative pod (DNS/network) | Session create/resume fails | Redis deployed in same namespace; smoke test validates connectivity end-to-end. | -| `containerConcurrency: 1` limits throughput | Slow under load | Acceptable for PoC; `max-scale: 5` allows 5 concurrent sessions. Production tuning is out of scope. | +| Risk | Impact | Mitigation | +| ---------------------------------------------------------------------------- | --------------------------- | --------------------------------------------------------------------------------------------------------------------------------- | +| Cold-start latency too high (Knative + Node startup + Redis read + LLM call) | Poor interactive UX | M4 measures but does not optimize — compaction-checkpoint (M5) addresses this. Acceptable for PoC. | +| Knative idle timeout too aggressive (kills pod mid-LLM-call) | Dropped request | `timeoutSeconds: 300` on the Knative revision gives LLM calls up to 5 minutes. | +| kubectl in-container cannot reach sandbox pod (RBAC) | Sandbox tools fail | Setup script creates a ServiceAccount with pod exec permissions; fallback: run without sandbox (tools are inert without env var). | +| Pi-fork build fragility (build order, version drift) | Dockerfile breaks | Pinned pi-fork commit (submodule); explicit ordered build; CI will catch drift. | +| Redis not reachable from Knative pod (DNS/network) | Session create/resume fails | Redis deployed in same namespace; smoke test validates connectivity end-to-end. | +| `containerConcurrency: 1` limits throughput | Slow under load | Acceptable for PoC; `max-scale: 5` allows 5 concurrent sessions. Production tuning is out of scope. | --- @@ -440,4 +444,4 @@ scale-to-zero works, M5 optimizes the cold-start path so it stays fast at scale. --- -*Assisted-By: Claude (Anthropic AI) * +_Assisted-By: Claude (Anthropic AI) _ diff --git a/docs/specs/2026-06-18-m10-mcp-code-mode-design.md b/docs/specs/2026-06-18-m10-mcp-code-mode-design.md index b8c8f8a..9ac61dc 100644 --- a/docs/specs/2026-06-18-m10-mcp-code-mode-design.md +++ b/docs/specs/2026-06-18-m10-mcp-code-mode-design.md @@ -10,7 +10,7 @@ Parent design: [Zero-Trust, Multi-Agent Extensions to the Serverless Harness](.. Builds on: M1 (Redis session backend), M2 (`K8sSandboxClient`), M3 (persistent channel), M4 (Knative wrapper) > **Supersedes §3.3 of the parent design.** The parent doc chose a dedicated MCP gateway -> with harness-registered `registerTool` *forwarding tools*. This spec replaces that with a +> with harness-registered `registerTool` _forwarding tools_. This spec replaces that with a > **code-execution-with-MCP** model: the model writes a script that calls MCP from inside the > sandbox, and credential injection + per-call audit happen transparently at the egress > waypoint. The spine (§2 of the parent) is unchanged and, if anything, more cleanly honored. @@ -36,13 +36,13 @@ The parent design settled three things about MCP and left one open: A verification finding in the parent design is decisive here: at the pinned Pi commit (`406a2214`) **Pi has no MCP client** — no `@modelcontextprotocol/sdk`, no MCP protocol -code. MCP integration must therefore be *built* regardless of locus, so "fits Pi as-is" +code. MCP integration must therefore be _built_ regardless of locus, so "fits Pi as-is" does not favor any option. Freed from that pull, we choose the locus that is **thinnest on Pi, most token-efficient, and most consistent with the spine**: the sandbox, invoked as code. -**The principle:** MCP is not a protocol the harness speaks. It is *code the model runs in -the hands*. This is a thin specialization of spine §3.2 ("the sandbox makes credentialed +**The principle:** MCP is not a protocol the harness speaks. It is _code the model runs in +the hands_. This is a thin specialization of spine §3.2 ("the sandbox makes credentialed egress calls it cannot read") — an MCP call is just another such egress. M10 is **done** when: @@ -72,7 +72,7 @@ M10 is **done** when: - **Budget-sum widening** in the existing M4-era budget voter to include broker-written MCP entries (the tunable soft budget). - **Recording** the credential/identity model (§5) that the egress path depends on. The - *implementation* of the identity plane (delegation issuance, offline grants) is M7–M9. + _implementation_ of the identity plane (delegation issuance, offline grants) is M7–M9. ### Out of scope (later milestones / separate tracks) @@ -87,25 +87,25 @@ M10 is **done** when: kagenti's `mcp-gateway` component is not on the sandbox→backend data path in this design. - **Subagent fan-out mechanics** — owned by M11; M10 only confirms MCP and the credential model compose with it (§9). -- **Env-injected credential broker** (OpenShell-style, for git/CLI tokens) — a *complement* +- **Env-injected credential broker** (OpenShell-style, for git/CLI tokens) — a _complement_ to the primary OBO path, pulled in only if/when M9 needs a token in the sandbox env (§5.4). --- ## 2. Key decisions -| # | Decision | Choice | -|---|----------|--------| -| D1 | Invocation locus | **Sandbox, as code.** The model uses Pi's existing Bash/Write/Read tools (already redirected to the pod by `K8sSandboxClient`, M2) to author and run a script that calls MCP. Full replacement of the §3.3 forwarding-tool approach — the brain never originates MCP. | -| D2 | Tool discovery | **Pre-baked into the sandbox image.** Interface stubs live on disk as `./servers//.ts`; the model reads only what it needs (progressive disclosure = the token win). Zero runtime codegen; reproducible. Trade-off: static roster per image. | -| D3 | Observability granularity | **Egress waypoint logs each call.** From Pi's view a bash run is one `intention` → one `tool_result`. The waypoint additionally appends one **per-MCP-call** entry into the *same session stream*, correlated by session id + `actor_spiffe_id`. The egress plane becomes a log *producer*. | -| D4 | Budget enforcement | **Lean both.** Waypoint enforces a high **hard** per-session call cap (kill-switch for runaway loops). Harness **soft** budget = the existing turn-boundary voter, with its cumulative sum widened to include broker-written MCP entries → clean `abort`. | -| D5 | Network / trust path | **Direct to backends + transparent waypoint.** Wrappers address real MCP server hostnames; the AuthBridge Envoy egress waypoint (Envoy + ext-proc) transparently intercepts at L7, injects creds, logs, and enforces the cap. No separate terminating gateway component. | -| D6 | Session correlation | **Runtime auto-stamps it.** The pre-baked client runtime reads `KAGENTI_SESSION_ID` from env (set by the harness when it binds the sandbox to a turn) and stamps `X-Kagenti-Session` on every MCP call. The model's code never threads session context. This correlation does **double duty**: per-call audit (D3) **and** resolving the bound user-subject for credential scoping (D9). | -| D7 | Pi integration surface | **Near-zero.** No `@modelcontextprotocol/sdk`, no `registerTool` MCP forwarding tools, no Pi fork for MCP. Reuses native `BashOperations`/`WriteOperations`/`ReadOperations`. | -| D8 | Credential broker base | **AuthBridge proxy.** It is the only candidate that already composes SPIFFE workload identity with user identity through an RFC 8693 token-exchange engine and does transparent L7 injection (matching D5). OpenShell's credentials-driver is a complement for env-injected creds only; DAM is a directional reference, not a buildable base (§5.5). | -| D9 | Per-user scoping | **SPIFFE actor + bound subject + pre-authorized delegation.** Inject keyed on *(session's bound user-subject ⊕ requesting workload's SPIFFE id)*. The agent's JWT-SVID is the RFC 8693 `actor_token`; `subject = bob`; a delegation grant stored in Keycloak authorizes that workload to act for that subject. Output: a fresh, short-lived, user+audience-scoped backend token. | -| D10 | Unattended sessions | **No live user token at runtime.** Because the user is offline, the subject is supplied from a **pre-authorized delegation / offline grant** held in the identity plane (Keycloak), not from a live inbound `Authorization` header. This is precisely the RFC 8693 §4.1 actor-token chaining AuthBridge lists as *not yet wired* — the M7/M8 build. | +| # | Decision | Choice | +| --- | ------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| D1 | Invocation locus | **Sandbox, as code.** The model uses Pi's existing Bash/Write/Read tools (already redirected to the pod by `K8sSandboxClient`, M2) to author and run a script that calls MCP. Full replacement of the §3.3 forwarding-tool approach — the brain never originates MCP. | +| D2 | Tool discovery | **Pre-baked into the sandbox image.** Interface stubs live on disk as `./servers//.ts`; the model reads only what it needs (progressive disclosure = the token win). Zero runtime codegen; reproducible. Trade-off: static roster per image. | +| D3 | Observability granularity | **Egress waypoint logs each call.** From Pi's view a bash run is one `intention` → one `tool_result`. The waypoint additionally appends one **per-MCP-call** entry into the _same session stream_, correlated by session id + `actor_spiffe_id`. The egress plane becomes a log _producer_. | +| D4 | Budget enforcement | **Lean both.** Waypoint enforces a high **hard** per-session call cap (kill-switch for runaway loops). Harness **soft** budget = the existing turn-boundary voter, with its cumulative sum widened to include broker-written MCP entries → clean `abort`. | +| D5 | Network / trust path | **Direct to backends + transparent waypoint.** Wrappers address real MCP server hostnames; the AuthBridge Envoy egress waypoint (Envoy + ext-proc) transparently intercepts at L7, injects creds, logs, and enforces the cap. No separate terminating gateway component. | +| D6 | Session correlation | **Runtime auto-stamps it.** The pre-baked client runtime reads `KAGENTI_SESSION_ID` from env (set by the harness when it binds the sandbox to a turn) and stamps `X-Kagenti-Session` on every MCP call. The model's code never threads session context. This correlation does **double duty**: per-call audit (D3) **and** resolving the bound user-subject for credential scoping (D9). | +| D7 | Pi integration surface | **Near-zero.** No `@modelcontextprotocol/sdk`, no `registerTool` MCP forwarding tools, no Pi fork for MCP. Reuses native `BashOperations`/`WriteOperations`/`ReadOperations`. | +| D8 | Credential broker base | **AuthBridge proxy.** It is the only candidate that already composes SPIFFE workload identity with user identity through an RFC 8693 token-exchange engine and does transparent L7 injection (matching D5). OpenShell's credentials-driver is a complement for env-injected creds only; DAM is a directional reference, not a buildable base (§5.5). | +| D9 | Per-user scoping | **SPIFFE actor + bound subject + pre-authorized delegation.** Inject keyed on _(session's bound user-subject ⊕ requesting workload's SPIFFE id)_. The agent's JWT-SVID is the RFC 8693 `actor_token`; `subject = bob`; a delegation grant stored in Keycloak authorizes that workload to act for that subject. Output: a fresh, short-lived, user+audience-scoped backend token. | +| D10 | Unattended sessions | **No live user token at runtime.** Because the user is offline, the subject is supplied from a **pre-authorized delegation / offline grant** held in the identity plane (Keycloak), not from a live inbound `Authorization` header. This is precisely the RFC 8693 §4.1 actor-token chaining AuthBridge lists as _not yet wired_ — the M7/M8 build. | --- @@ -147,7 +147,7 @@ returns as ONE bash tool_result to the model context ← the token win **Why this shape.** The large token reduction reported for code-execution-with-MCP comes from two properties this path preserves: (1) tool definitions are files on disk read on demand, never a context dump; (2) intermediate MCP results are processed in code and never -enter the model context — only the final slice does. Pi is a *coding* agent, so +enter the model context — only the final slice does. Pi is a _coding_ agent, so writing-and-running-code is its strongest mode; this plays to its grain rather than bolting an RPC tool surface onto it. @@ -155,12 +155,12 @@ an RPC tool surface onto it. ## 4. Components -| Component | Where it runs | Responsibility | -|---|---|---| -| **MCP wrapper library + client runtime** | pre-baked in the sandbox image | `./servers//.ts` interface stubs (schemas only — no creds, no live endpoints baked in) + a thin MCP-over-HTTP client. Runtime auto-stamps `X-Kagenti-Session` from `KAGENTI_SESSION_ID`. | -| **AuthBridge waypoint MCP extension** | egress path of the sandbox pod | Recognize MCP-over-HTTP; resolve session→subject; perform the RFC 8693 delegation exchange; inject the user-scoped token; append a per-call log entry; count calls + enforce the hard cap. | -| **Identity plane (M7–M9)** | cluster | SPIRE issues SPIFFE SVIDs to harness/sandbox/subagents; Keycloak holds OAuth clients, per-user pre-authorized delegations / offline grants, and performs token exchange. **M10 consumes this; it does not build it.** | -| **Budget-sum widening** | harness (existing M4-era voter) | The turn-boundary cumulative sum now also counts the broker-written MCP entries. A few lines; no new mechanism. | +| Component | Where it runs | Responsibility | +| ---------------------------------------- | ------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| **MCP wrapper library + client runtime** | pre-baked in the sandbox image | `./servers//.ts` interface stubs (schemas only — no creds, no live endpoints baked in) + a thin MCP-over-HTTP client. Runtime auto-stamps `X-Kagenti-Session` from `KAGENTI_SESSION_ID`. | +| **AuthBridge waypoint MCP extension** | egress path of the sandbox pod | Recognize MCP-over-HTTP; resolve session→subject; perform the RFC 8693 delegation exchange; inject the user-scoped token; append a per-call log entry; count calls + enforce the hard cap. | +| **Identity plane (M7–M9)** | cluster | SPIRE issues SPIFFE SVIDs to harness/sandbox/subagents; Keycloak holds OAuth clients, per-user pre-authorized delegations / offline grants, and performs token exchange. **M10 consumes this; it does not build it.** | +| **Budget-sum widening** | harness (existing M4-era voter) | The turn-boundary cumulative sum now also counts the broker-written MCP entries. A few lines; no new mechanism. | **Wrappers ≠ credentials/endpoints.** Pre-baking bakes only interface stubs. The live target URL and credentials are resolved at runtime through the egress plane, so the same image is @@ -169,20 +169,20 @@ portable across environments and pre-baking never violates the cred spine. **What binds a sandbox to a session — and to a user.** The harness sets `KAGENTI_SESSION_ID` (and the sandbox receives its SPIFFE id from SPIRE) when it provisions/binds the sandbox for a turn. The session record carries the initiating user's **subject reference** (e.g. `bob`). -That env + SPIFFE id + bound subject is what lets the *transparent* waypoint both correlate +That env + SPIFFE id + bound subject is what lets the _transparent_ waypoint both correlate nested MCP calls to the right session stream **and** mint the right user-scoped token. -**Delta vs. parent §3.3.** This design *removes* a Pi integration (the forwarding tools) and -*replaces* the standalone MCP gateway component with an extension to a waypoint kagenti +**Delta vs. parent §3.3.** This design _removes_ a Pi integration (the forwarding tools) and +_replaces_ the standalone MCP gateway component with an extension to a waypoint kagenti already runs. --- ## 5. Credential & identity model -This section records the model the egress path depends on. The *engine* (token exchange, -SPIFFE issuance) largely exists in AuthBridge/SPIRE/Keycloak today; the *unattended -delegation* piece is the M7–M9 build. Recorded here so those milestones inherit a settled +This section records the model the egress path depends on. The _engine_ (token exchange, +SPIFFE issuance) largely exists in AuthBridge/SPIRE/Keycloak today; the _unattended +delegation_ piece is the M7–M9 build. Recorded here so those milestones inherit a settled shape. ### 5.1 Where credentials live (and don't) @@ -190,12 +190,12 @@ shape. Both kagenti credential subsystems converge on the spine's "broker is the sole secret holder" property: -| Layer | AuthBridge | OpenShell credentials-keycloak | -|---|---|---| -| **At rest** | Keycloak OAuth clients (client_id+secret, or SPIFFE JWT-SVID assertion) + K8s Secret mounted into the sidecar | K8s Secret files mounted into the credentials-driver sidecar | -| **At runtime** | in-memory in the AuthBridge **sidecar** only | in-memory in the credentials **driver** sidecar only | -| **Harness / sandbox** | never hold raw secrets | never hold raw secrets | -| **Injection** | L7 `Authorization` header rewrite (Envoy / ext-proc), transparent | short-lived token → `DriverSandboxSpec.environment` env var, via UDS gRPC | +| Layer | AuthBridge | OpenShell credentials-keycloak | +| --------------------- | ------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------- | +| **At rest** | Keycloak OAuth clients (client_id+secret, or SPIFFE JWT-SVID assertion) + K8s Secret mounted into the sidecar | K8s Secret files mounted into the credentials-driver sidecar | +| **At runtime** | in-memory in the AuthBridge **sidecar** only | in-memory in the credentials **driver** sidecar only | +| **Harness / sandbox** | never hold raw secrets | never hold raw secrets | +| **Injection** | L7 `Authorization` header rewrite (Envoy / ext-proc), transparent | short-lived token → `DriverSandboxSpec.environment` env var, via UDS gRPC | **Answer:** credentials live in **Keycloak + a K8s Secret, loaded into the broker sidecar's memory; nowhere else.** The session log carries only **identity references** (subject SPIFFE @@ -208,22 +208,22 @@ live user JWT after the idle gap, so the user dimension cannot come from a live token. It comes from a **pre-authorized delegation**: - **SPIFFE identity = the requester's right to act.** The session/sandbox/subagent each - carry a JWT-SVID. At egress that SVID is the RFC 8693 `actor_token` — it proves *which - workload* is asking. + carry a JWT-SVID. At egress that SVID is the RFC 8693 `actor_token` — it proves _which + workload_ is asking. - **User identity = the subject acted for.** The session is bound to `subject = bob`, an identity reference in the log (never a secret). - **The grant that makes it unattended** = a pre-authorized delegation stored in Keycloak ("this agent workload identity may act on behalf of `bob` for audience X"), established once when Bob consents. No live user token is needed at runtime. -At egress the broker presents *actor = agent JWT-SVID, subject = bob, audience = backend* → +At egress the broker presents _actor = agent JWT-SVID, subject = bob, audience = backend_ → Keycloak validates the stored delegation → mints a fresh, short-lived, user+audience-scoped token → injects it. Bob's sessions get Bob-scoped tokens while Bob is offline; Alice's get Alice-scoped. **Blast radius:** a compromised sandbox in Bob's session can only ever act as Bob, because the actor SVID + bound subject are what authorize the mint. > **Concrete finding:** the mechanism above is exactly the RFC 8693 **actor-token / -> delegation chaining (§4.1)** that AuthBridge today lists as *not wired yet*. The AuthBridge +> delegation chaining (§4.1)** that AuthBridge today lists as _not wired yet_. The AuthBridge > TODO **is** the unattended-delegation feature. Building it is additive to an engine that > already exists, and is the substance of the M7/M8 spine work. @@ -233,13 +233,13 @@ AuthBridge's `2026-06-02-credential-placeholder-swap-design.md` carries an **opa in the request/conversation and swaps in the real token at egress in the sidecar. We adopt it because it defends both invariant §4.1 (no secret in the log) **and** the prompt-injection threat (no secret in model-visible output): the sandbox's MCP client never even holds a -placeholder *value* it could leak — at most a session header — and the sidecar supplies the +placeholder _value_ it could leak — at most a session header — and the sidecar supplies the token. ### 5.4 Env-injected credentials (complement, deferred) -MCP-over-HTTP is served by L7 header injection. Some sandbox operations need a token *in -env* for a CLI — notably `git push` (parent M9). For those, the **OpenShell +MCP-over-HTTP is served by L7 header injection. Some sandbox operations need a token _in +env_ for a CLI — notably `git push` (parent M9). For those, the **OpenShell credentials-driver pattern** fits: a sidecar broker over a Unix-domain socket, `ResolveCredential(name, subject) → short-lived token`, injected as an env var, with the raw secret never leaving the sidecar. This is pulled in **only if M9 needs it**; one @@ -248,20 +248,20 @@ user-subject binding (§5.2) feeds both paths. It is a weaker identity model on ### 5.5 Best-fit comparison (why AuthBridge is the base) -For *unattended + SPIFFE + user identity*: +For _unattended + SPIFFE + user identity_: -| | AuthBridge proxy | OpenShell gateway / credentials-driver | DAM | -|---|---|---|---| -| **SPIFFE workload identity** | ✅ native (JWT-SVID client assertion) | ❌ not in cred path (static `client_credentials`) | ❓ undocumented | -| **User identity in cred mint** | ✅ as `subject_token` (OBO) | ❌ `user_id` not passed to driver (their TODO) | ✅ "their own credentials" (per README) | -| **Token exchange (RFC 8693)** | ✅ engine exists | ❌ none | ❓ | -| **Unattended / offline delegation** | ⚠️ needs actor-token chaining built (the §4.1 TODO) | ⚠️ needs offline grant + user plumbing built | ✅ "runs after you close your laptop" — internals SSO-gated | -| **Transparent L7 inject (MCP-over-HTTP, D5)** | ✅ header rewrite | ➖ env injection (CLI-shaped) | ❓ | -| **Secret-out-of-log** | ✅ placeholder-swap design | ✅ token never leaves sidecar | ✅ "no creds in runtime" | -| **Verifiable / buildable now** | ✅ local source | ✅ local source | ❌ SSO-gated, ACP-not-MCP | +| | AuthBridge proxy | OpenShell gateway / credentials-driver | DAM | +| --------------------------------------------- | --------------------------------------------------- | ------------------------------------------------- | ----------------------------------------------------------- | +| **SPIFFE workload identity** | ✅ native (JWT-SVID client assertion) | ❌ not in cred path (static `client_credentials`) | ❓ undocumented | +| **User identity in cred mint** | ✅ as `subject_token` (OBO) | ❌ `user_id` not passed to driver (their TODO) | ✅ "their own credentials" (per README) | +| **Token exchange (RFC 8693)** | ✅ engine exists | ❌ none | ❓ | +| **Unattended / offline delegation** | ⚠️ needs actor-token chaining built (the §4.1 TODO) | ⚠️ needs offline grant + user plumbing built | ✅ "runs after you close your laptop" — internals SSO-gated | +| **Transparent L7 inject (MCP-over-HTTP, D5)** | ✅ header rewrite | ➖ env injection (CLI-shaped) | ❓ | +| **Secret-out-of-log** | ✅ placeholder-swap design | ✅ token never leaves sidecar | ✅ "no creds in runtime" | +| **Verifiable / buildable now** | ✅ local source | ✅ local source | ❌ SSO-gated, ACP-not-MCP | **Decision (D8):** AuthBridge proxy as the base — the only candidate that already composes -SPIFFE workload identity with user identity via a token-exchange engine *and* does the +SPIFFE workload identity with user identity via a token-exchange engine _and_ does the transparent L7 injection the M10 MCP-over-HTTP path needs. The single missing capability is sourcing the subject from a pre-authorized delegation (D10) rather than a live inbound token — additive. OpenShell's credentials-driver is the right complement for env creds (§5.4); DAM @@ -277,12 +277,12 @@ is a directional reference only (SSO-gated, ACP not MCP, SPIFFE usage unverifiab - **Coarse pair (harness-produced):** the model's bash run is one `intention` (the script) → one `tool_result` (the printed slice) — identical to any other bash call. - **Fine entries (broker-produced):** the waypoint appends one entry per MCP call into the - *same session stream*, carrying `data.via = "mcp-waypoint"`, the target `server`/`tool`, + _same session stream_, carrying `data.via = "mcp-waypoint"`, the target `server`/`tool`, `actor_spiffe_id`, and the bound `subject` reference. **No new entry type** is required — - but the egress plane is now a log *producer*, not only the harness. + but the egress plane is now a log _producer_, not only the harness. **Invariant §4.1 holds unchanged:** no entry carries a raw secret. Credentials are injected -downstream of the log, at the waypoint; entries name *who / what / where / on-behalf-of-whom* +downstream of the log, at the waypoint; entries name _who / what / where / on-behalf-of-whom_ — never the credential. This stays directly testable by the parent design's red-team grep over the log, the harness env, the sandbox env, and the reconstructed conversation. @@ -299,21 +299,21 @@ over the log, the harness env, the sandbox env, and the reconstructed conversati broker's MCP entries. This is the operator-tunable budget and produces a clean `abort`. **Why both, despite a simplicity preference.** Code-mode introduces one genuinely new failure -mode: the model writes a loop that makes thousands of MCP calls inside a *single* bash run. +mode: the model writes a loop that makes thousands of MCP calls inside a _single_ bash run. No inference happens during that run, so the inference budget never sees it, and a pure -turn-boundary budget only reacts *after* the turn. The hard cap closes that intra-turn hole. +turn-boundary budget only reacts _after_ the turn. The hard cap closes that intra-turn hole. Because both halves are small additions to code paths we already build (D3 logging path; M4 voter), "both" costs marginally more than one and avoids having to choose between ugly -mid-script failures *or* an unguarded runaway path. +mid-script failures _or_ an unguarded runaway path. **Failure modes (new, stated explicitly):** -- **Hard-cap hit** → the MCP call throws *inside the model's script* → the model sees an +- **Hard-cap hit** → the MCP call throws _inside the model's script_ → the model sees an error mid-run and adapts. Acceptable: errors-in-scripts are a coding agent's native failure surface. - **Single-turn budget burn** → a bad script can still exhaust a turn's soft budget in one - bash call; the soft budget catches it at the *next* turn boundary, the hard cap catches - *pathological* loops mid-turn. + bash call; the soft budget catches it at the _next_ turn boundary, the hard cap catches + _pathological_ loops mid-turn. - **Delegation expired / revoked** → the exchange fails at the waypoint; the MCP call returns an auth error into the script. The model surfaces it; the session does not silently fall back to a less-scoped or workload-only token. @@ -326,7 +326,7 @@ This is the crux of the original "how does this work with Pi?" question. The ans **near-zero surface — less than §3.3 required.** - ✗ No `@modelcontextprotocol/sdk` dependency. -- ✗ No `registerTool` MCP forwarding tools (§3.3 *did* need these). +- ✗ No `registerTool` MCP forwarding tools (§3.3 _did_ need these). - ✗ No Pi fork for MCP. - ✓ Reuses Pi's existing `BashOperations` / `WriteOperations` / `ReadOperations`, already redirected to the pod by `K8sSandboxClient` (M2). @@ -356,7 +356,7 @@ in M11 for this to hold. "reachable only through the mediating waypoint." Isolation now comes from the mesh + NetworkPolicy + the ext-proc, not from an air-gap. For high-risk untrusted backends this is weaker than a terminating proxy; if that becomes unacceptable, a terminating - `mcp-gateway` hop can be reintroduced for *specific* backends without changing the + `mcp-gateway` hop can be reintroduced for _specific_ backends without changing the code-mode model (the wrappers' target hostname simply points at the gateway for those). 2. **Static roster per image.** Rebuild the sandbox image when the MCP server set changes; per-team/per-session rosters need image variants (deferred, §1 out of scope). @@ -411,4 +411,4 @@ extension, and the M7–M9 identity plane deployed: --- -*Assisted-By: Claude (Anthropic AI) * +_Assisted-By: Claude (Anthropic AI) _ diff --git a/docs/specs/2026-06-19-m13-generalized-credentialed-egress-design.md b/docs/specs/2026-06-19-m13-generalized-credentialed-egress-design.md index 541d88f..2be70df 100644 --- a/docs/specs/2026-06-19-m13-generalized-credentialed-egress-design.md +++ b/docs/specs/2026-06-19-m13-generalized-credentialed-egress-design.md @@ -13,13 +13,13 @@ Sibling / specializes: [M10 — MCP via Code-Mode in the Sandbox](2026-06-18-m10 Builds on: M1 (Redis session backend), M2 (`K8sSandboxClient`), M3 (persistent channel), M4 (Knative wrapper), M10 (MCP code-mode + credential/identity model §5, placeholder-swap §5.3) > **Relationship to M10.** M10 is frozen and approved. This design does not reopen it; it -> *generalizes the credential mechanism M10 already adopted* — the **placeholder-swap** pattern +> _generalizes the credential mechanism M10 already adopted_ — the **placeholder-swap** pattern > (M10 §5.3) — to all HTTP egress. The selector, the resolvers, the audit producer, and the > budget machinery are shared. Only the **interception point** differs: M10's in-mesh waypoint > for in-mesh MCP backends; a **forward proxy** for external hosts. > **Changelog 2.0 (supersedes v1.0).** v1.0 chose an in-mesh front-door + host-based injection -> (model sends no auth; proxy adds it) explicitly to *avoid* TLS interception, at the cost of +> (model sends no auth; proxy adds it) explicitly to _avoid_ TLS interception, at the cost of > per-host mesh config, a generic egress helper, per-API wrappers for correctness, and an > env-injection escape hatch for foreign binaries. v2.0 **pivots to AuthBridge's own > placeholder-swap pattern over a forward proxy**: the sandbox/tools hold only inert @@ -50,7 +50,7 @@ enforces a cap. The sandbox, the prompt, and the log never see a real secret. ### Worked example (the shape of "done") -A user prompts: *"review the PR at https://github.com/kagenti/kagenti/pull/1990"* — and knows +A user prompts: _"review the PR at https://github.com/kagenti/kagenti/pull/1990"_ — and knows nothing about proxies, placeholders, or credentials. 1. The model maps intent → GitHub and writes a sandbox script the natural way, e.g. @@ -61,7 +61,7 @@ nothing about proxies, placeholders, or credentials. session → `subject = `, host-policy → the user's stored GitHub grant, **overwrites** the header with the real token, originates real TLS to api.github.com, appends one audit entry, counts it. -3. GitHub sees the call **as that user**. The script filters/summarizes the diff *in code* and +3. GitHub sees the call **as that user**. The script filters/summarizes the diff _in code_ and prints only the review-relevant slice → one `tool_result`. A 3000-line PR never floods context — the filter-in-code token win, preserved. @@ -77,16 +77,16 @@ nothing about proxies, placeholders, or credentials. - **Generalized audit + budget:** per-call `egress-broker` log entries (full L7); hard per-session egress-call cap; soft turn-boundary budget widened to count them. - **Recording** the per-user external-credential-store + linking/consent shape (§4) as a hard - identity-plane dependency this milestone *consumes*, mirroring M10 §5. + identity-plane dependency this milestone _consumes_, mirroring M10 §5. ### Out of scope (later milestones / separate tracks) - **The per-user external-credential store + onboarding/consent build itself** — M7–M9 identity-plane work. This design records its required shape; it does not build it. - **Optional progressive-disclosure wrappers.** Native tools are first-class (§5); pre-baked - per-API wrappers are a *later, optional* token-optimization for very large API surfaces, not + per-API wrappers are a _later, optional_ token-optimization for very large API surfaces, not required for correctness. Not built here. -- **Request-signing APIs (AWS SigV4 and similar).** Where the secret *signs* the request, a +- **Request-signing APIs (AWS SigV4 and similar).** Where the secret _signs_ the request, a post-hoc header swap is impossible; these stay an env-inject-real-key escape hatch (§5.3), deferred. - **Per-team / per-session host rosters.** Static per environment; deferred. @@ -97,18 +97,18 @@ nothing about proxies, placeholders, or credentials. ## 2. Key decisions -| # | Decision | Choice | -|---|----------|--------| -| E1 | Unification | **One credential mechanism across all egress; MCP is one interception case.** Shared selector, resolvers, placeholder-swap, audit producer, and budget. Interception differs only by locality: in-mesh waypoint (MCP/internal, M10) vs. forward proxy (external). | -| E2 | Credential presentation | **Placeholder-swap (AuthBridge-native).** The sandbox/tool holds only an **inert placeholder**; the proxy overwrites it with the real resolved credential at egress. Tools use their native auth mechanism unchanged. | -| E3 | External interception | **Forward proxy (`HTTPS_PROXY`) + baked CA** in the sandbox trust store. TLS interception is accepted and is the enabling mechanism. The CA is a **trust anchor, not a secret** (the proxy's private key never leaves the proxy). | -| E4 | Egress allowlist | **Enforced at the proxy.** It forwards only to allowlisted hosts; the host roster is the allowlist and the exfil boundary. "curl to anywhere" is structurally impossible. | -| E5 | Non-OAuth scoping | **Per-user stored credentials. No shared-workload fallback.** Non-mintable destinations (GitHub PAT/App, third-party keys) resolve to the user's own pre-linked grant, keyed by bound subject. | -| E6 | Credential resolver | **Two resolvers, one selector.** `(subject ⊕ destination)` resolves via **mint (RFC 8693)** for OAuth backends (M10 path) or **fetch-stored-grant** for everything else. The host-policy entry declares which, and which request field carries the credential. | -| E7 | Model interface | **Native tools, first-class.** `gh`/`curl`/SDKs work unmodified using placeholder creds from env/config — the model writes the code a human would. Wrappers + a generic helper drop to **optional** token-optimization, not required for correctness or steering. The filter-in-code token win is preserved by code-mode regardless. | -| E8 | Escape hatch | **Only request-signing schemes** (AWS SigV4 etc.), where the secret signs the request and cannot be swapped post-hoc, remain an env-inject-real-key escape hatch. Bearer/OAuth/header APIs (incl. GitHub) are first-class. | -| E9 | Security boundary | **The forward proxy is the sole egress + enforcement point** — allowlist, resolve, overwrite, audit, cap. Sandbox code is fully bypass-capable and enforces nothing. | -| E10 | Home | **New sibling milestone (this doc).** Keeps M10 frozen; records the new identity-plane dependency rather than entangling timelines. | +| # | Decision | Choice | +| --- | ----------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| E1 | Unification | **One credential mechanism across all egress; MCP is one interception case.** Shared selector, resolvers, placeholder-swap, audit producer, and budget. Interception differs only by locality: in-mesh waypoint (MCP/internal, M10) vs. forward proxy (external). | +| E2 | Credential presentation | **Placeholder-swap (AuthBridge-native).** The sandbox/tool holds only an **inert placeholder**; the proxy overwrites it with the real resolved credential at egress. Tools use their native auth mechanism unchanged. | +| E3 | External interception | **Forward proxy (`HTTPS_PROXY`) + baked CA** in the sandbox trust store. TLS interception is accepted and is the enabling mechanism. The CA is a **trust anchor, not a secret** (the proxy's private key never leaves the proxy). | +| E4 | Egress allowlist | **Enforced at the proxy.** It forwards only to allowlisted hosts; the host roster is the allowlist and the exfil boundary. "curl to anywhere" is structurally impossible. | +| E5 | Non-OAuth scoping | **Per-user stored credentials. No shared-workload fallback.** Non-mintable destinations (GitHub PAT/App, third-party keys) resolve to the user's own pre-linked grant, keyed by bound subject. | +| E6 | Credential resolver | **Two resolvers, one selector.** `(subject ⊕ destination)` resolves via **mint (RFC 8693)** for OAuth backends (M10 path) or **fetch-stored-grant** for everything else. The host-policy entry declares which, and which request field carries the credential. | +| E7 | Model interface | **Native tools, first-class.** `gh`/`curl`/SDKs work unmodified using placeholder creds from env/config — the model writes the code a human would. Wrappers + a generic helper drop to **optional** token-optimization, not required for correctness or steering. The filter-in-code token win is preserved by code-mode regardless. | +| E8 | Escape hatch | **Only request-signing schemes** (AWS SigV4 etc.), where the secret signs the request and cannot be swapped post-hoc, remain an env-inject-real-key escape hatch. Bearer/OAuth/header APIs (incl. GitHub) are first-class. | +| E9 | Security boundary | **The forward proxy is the sole egress + enforcement point** — allowlist, resolve, overwrite, audit, cap. Sandbox code is fully bypass-capable and enforces nothing. | +| E10 | Home | **New sibling milestone (this doc).** Keeps M10 frozen; records the new identity-plane dependency rather than entangling timelines. | | E11 | Placeholder semantics & exfil | **Placeholders are inert markers** that satisfy client tools. The proxy overwrites with the real credential **only for the matching allowlisted destination**, keyed on `(subject ⊕ destination)`. A placeholder sent anywhere else is never swapped and the host isn't reachable — so a leaked placeholder is worthless and a real credential never leaves its bound destination. | --- @@ -151,7 +151,7 @@ writes arbitrary code, nothing in the sandbox can constrain reachability — all resolution, overwrite, audit, and cap all live at the proxy. **MCP stays M10.** In-mesh MCP backends keep M10's in-mesh waypoint interception (mesh mTLS, a -legitimate terminator — no CA needed there). M13 adds the *forward-proxy* interception for +legitimate terminator — no CA needed there). M13 adds the _forward-proxy_ interception for external hosts. Same brain (selector, resolvers, placeholder-swap, audit, cap), two interception points by locality. @@ -165,10 +165,10 @@ Every egress resolves `(bound subject ⊕ destination host) → credential` at t host-policy entry for the destination declares the resolver **and** which request field carries the credential (e.g. `Authorization` header, bearer): -| Resolver | When | How | Per-user? | -|---|---|---|---| -| **Mint (RFC 8693)** | destination speaks OAuth (M10 MCP backends, OIDC APIs) | actor = workload JWT-SVID, subject = ``, audience = backend → fresh scoped token | yes, minted | -| **Stored grant** | non-exchangeable API (GitHub PAT/App installation, third-party key) | fetch the user's pre-linked credential; refresh if supported | yes, stored | +| Resolver | When | How | Per-user? | +| ------------------- | ------------------------------------------------------------------- | -------------------------------------------------------------------------------------- | ----------- | +| **Mint (RFC 8693)** | destination speaks OAuth (M10 MCP backends, OIDC APIs) | actor = workload JWT-SVID, subject = ``, audience = backend → fresh scoped token | yes, minted | +| **Stored grant** | non-exchangeable API (GitHub PAT/App installation, third-party key) | fetch the user's pre-linked credential; refresh if supported | yes, stored | The minting resolver is M10 §5 verbatim. The stored-grant resolver is the new capability and the reason for the identity-plane dependency below. @@ -193,7 +193,7 @@ milestone **consumes**, does not build, the store): egress call returns an **auth error into the script** — **no silent fallback** to a workload-only or less-scoped credential. - **Interactive vs. unattended are identical at egress.** GitHub login is not part of kagenti's - Keycloak login, so even an online, interactive session reads the user's *stored* grant — not a + Keycloak login, so even an online, interactive session reads the user's _stored_ grant — not a live propagated GitHub token. Unattended (scale-to-zero, user offline) reads the same grant. This subsumes M10's live-or-delegated subject sourcing. @@ -214,7 +214,8 @@ milestone **consumes**, does not build, the store): `subject` and `actor_spiffe_id` are identity references, never secrets. Injection happens downstream of the log, at the proxy. The parent's red-team grep (log + harness env + sandbox env -+ reconstructed prompt) remains the direct test. + +- reconstructed prompt) remains the direct test. --- @@ -226,12 +227,12 @@ no front-door hostname, no endpoint-override, no special convention. Real hostna (`api.github.com`) are used directly; `HTTPS_PROXY` routes egress to the broker transparently. **The filter-in-code token win is preserved regardless.** Code-mode means the model's script -runs the tool, processes output *in code*, and prints only the relevant slice — so a huge PR +runs the tool, processes output _in code_, and prints only the relevant slice — so a huge PR diff or API response never floods context. This holds whether the script uses `gh`, `curl`, or a wrapper; it is a property of running tools inside a script, not of any wrapper. **Optional wrappers (deferred, not required).** For a very large API surface where even -*discovering* endpoints costs context, pre-baked `./apis//…` stubs can be added later as +_discovering_ endpoints costs context, pre-baked `./apis//…` stubs can be added later as a pure progressive-disclosure optimization. They are **not** needed for correctness or steering under v2.0 (native tools already work), and they are **never** a security boundary (§5.1). @@ -271,7 +272,7 @@ common case, GitHub included — never touch it. ## 6. Observability, budget & failure modes -**Audit (generalize M10 D3).** The proxy is the log *producer* for all egress; full L7 visibility +**Audit (generalize M10 D3).** The proxy is the log _producer_ for all egress; full L7 visibility (it terminates TLS) means per-call entries carry method + path: ``` @@ -288,7 +289,7 @@ the overwrite is downstream of the log. **Budget (M10's "lean both," generalized).** -- **Hard cap (proxy):** a high per-session *egress-call* ceiling — kill-switch for a runaway loop +- **Hard cap (proxy):** a high per-session _egress-call_ ceiling — kill-switch for a runaway loop (the model writes a loop making thousands of calls inside one bash run, where no inference happens and the turn-boundary budget never sees it). A counter + one comparison on the audit path. @@ -297,7 +298,7 @@ the overwrite is downstream of the log. **Failure modes:** -- **Hard-cap hit** → the egress call fails *inside the model's script*; the model adapts mid-run. +- **Hard-cap hit** → the egress call fails _inside the model's script_; the model adapts mid-run. - **Single-turn budget burn** → caught at the next turn boundary by the soft budget; pathological intra-turn loops caught by the hard cap. - **Grant expired / revoked** → the resolver fails at the proxy; an auth error returns into the @@ -351,7 +352,7 @@ M13 passes when, end to end on a Kind cluster with the sandbox image (baked CA + placeholders), the AuthBridge forward proxy, and the M7–M9 identity plane (including the per-user external-credential store) deployed: -1. *"review PR 1990"* completes purely by the model running a **native** sandbox script +1. _"review PR 1990"_ completes purely by the model running a **native** sandbox script (`gh`/`curl`); `grep` confirms no real token is present in the harness bundle, sandbox env (placeholder only), or prompt. 2. The real GitHub credential is present **only** at the proxy: red-team grep finds it absent from @@ -387,4 +388,4 @@ external-credential store) deployed: --- -*Assisted-By: Claude (Anthropic AI) * +_Assisted-By: Claude (Anthropic AI) _ diff --git a/docs/specs/2026-06-23-m5-compaction-checkpoint-design.md b/docs/specs/2026-06-23-m5-compaction-checkpoint-design.md index d4f1f35..90f002c 100644 --- a/docs/specs/2026-06-23-m5-compaction-checkpoint-design.md +++ b/docs/specs/2026-06-23-m5-compaction-checkpoint-design.md @@ -54,7 +54,7 @@ M5 is **done** when: ### Honest reframing (right-sizes the milestone) -The parent plan's premise was: "without a checkpoint, cold start replays *all* turns to the +The parent plan's premise was: "without a checkpoint, cold start replays _all_ turns to the LLM, so latency grows with session length." Discovery (file:line evidence in §2) shows this is **already false for Pi** — Pi compaction is a first-class log entry, and `buildSessionContext()` (`session-manager.ts:330–432`) already assembles only `[compaction summary, …kept tail, @@ -63,7 +63,7 @@ compaction. What remains O(total entries) on cold start is purely **local** reconstruction: the Redis read, the `_buildIndex()` pass, and the leaf→root walk in `buildSessionContext()`. The loader's -measurable win is bounding *those* to O(tail) — reduced Redis bandwidth and CPU at large +measurable win is bounding _those_ to O(tail) — reduced Redis bandwidth and CPU at large session lengths — **not** a reduction in LLM latency, which dominates wall-clock and is already bounded. This spec builds the loader as the parent plan's "O(1) reconstruction" deliverable while setting honest expectations: the parent plan's E2 experiment should measure **local @@ -97,33 +97,33 @@ reconstruction cost**, not end-to-end latency, to see the effect cleanly. All references are in `pi-fork/packages/coding-agent/src/`. -| # | Finding | Evidence | -|---|---------|----------| -| F1 | Pi persists compaction as a first-class log entry carrying the summary, `firstKeptEntryId`, and `tokensBefore`. | `SessionManager.appendCompaction(summary, firstKeptEntryId, tokensBefore, details?, fromHook?)` — `session-manager.ts:1009`; `CompactionEntry` — `session-manager.ts:70`. | -| F2 | `buildSessionContext()` reconstructs only `[summary, …kept tail, …post-compaction]` — never the full history. | `session-manager.ts:401–424`. | -| F3 | The leaf→root walk **stops gracefully when a parent is not in the loaded set** (`byId.get(parentId)` → `undefined` ends the loop). This is what makes tail-only loading safe. | `session-manager.ts:357–363`. | -| F4 | The kept messages sit in the log **before** the compaction entry, emitted from `firstKeptEntryId`. ⇒ the resume slice must start at `firstKeptEntryId`, not at the compaction entry's position. | `session-manager.ts:406–418`. | -| F5 | `SessionStorageBackend` already has `read(sid, fromPosition?)` and `latestCheckpoint()`; `latestCheckpoint` is **declared but never called by Pi core**, so we own its semantics. | `core/session-storage-backend.ts:14–22`; no other call site. | -| F6 | `openFromBackend` is the only public loader and reads **all** entries (`backend.read(sessionId)`, no `fromPosition`). `loadFromEntries`/`_buildIndex` are private. ⇒ a fork-free harness-side tail loader is not possible; the loader belongs in `SessionManager`. | `session-manager.ts:1422–1434`, `:829–834`. | -| F7 | `tool_call` can block (`{ block: true, reason }`) and mutate `event.input`. | `ToolCallEventResult` — `core/types.ts:1020`; emit — `agent-session.ts:403–423`; block — `runner.ts:875`. | -| F8 | Live token usage is available to handlers via `ctx.getContextUsage()` / `getSessionStats()`. `getContextUsage().tokens` is **`null`** between a compaction and the next LLM response. | `agent-session.ts:2923` (`getSessionStats`), `:2968` (`getContextUsage`); `ContextUsage` — `types.ts:281`. | -| F9 | `read(fromPosition)` today reads the whole stream then filters by position (O(N)); entries are appended with auto stream id `"*"`. | `packages/session-backend/src/redis-backend.ts:41–54`, `:31`. | +| # | Finding | Evidence | +| --- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| F1 | Pi persists compaction as a first-class log entry carrying the summary, `firstKeptEntryId`, and `tokensBefore`. | `SessionManager.appendCompaction(summary, firstKeptEntryId, tokensBefore, details?, fromHook?)` — `session-manager.ts:1009`; `CompactionEntry` — `session-manager.ts:70`. | +| F2 | `buildSessionContext()` reconstructs only `[summary, …kept tail, …post-compaction]` — never the full history. | `session-manager.ts:401–424`. | +| F3 | The leaf→root walk **stops gracefully when a parent is not in the loaded set** (`byId.get(parentId)` → `undefined` ends the loop). This is what makes tail-only loading safe. | `session-manager.ts:357–363`. | +| F4 | The kept messages sit in the log **before** the compaction entry, emitted from `firstKeptEntryId`. ⇒ the resume slice must start at `firstKeptEntryId`, not at the compaction entry's position. | `session-manager.ts:406–418`. | +| F5 | `SessionStorageBackend` already has `read(sid, fromPosition?)` and `latestCheckpoint()`; `latestCheckpoint` is **declared but never called by Pi core**, so we own its semantics. | `core/session-storage-backend.ts:14–22`; no other call site. | +| F6 | `openFromBackend` is the only public loader and reads **all** entries (`backend.read(sessionId)`, no `fromPosition`). `loadFromEntries`/`_buildIndex` are private. ⇒ a fork-free harness-side tail loader is not possible; the loader belongs in `SessionManager`. | `session-manager.ts:1422–1434`, `:829–834`. | +| F7 | `tool_call` can block (`{ block: true, reason }`) and mutate `event.input`. | `ToolCallEventResult` — `core/types.ts:1020`; emit — `agent-session.ts:403–423`; block — `runner.ts:875`. | +| F8 | Live token usage is available to handlers via `ctx.getContextUsage()` / `getSessionStats()`. `getContextUsage().tokens` is **`null`** between a compaction and the next LLM response. | `agent-session.ts:2923` (`getSessionStats`), `:2968` (`getContextUsage`); `ContextUsage` — `types.ts:281`. | +| F9 | `read(fromPosition)` today reads the whole stream then filters by position (O(N)); entries are appended with auto stream id `"*"`. | `packages/session-backend/src/redis-backend.ts:41–54`, `:31`. | --- ## 3. Key decisions -| # | Decision | Choice | -|---|----------|--------| -| D1 | What is "the checkpoint"? | **Pi's native `compaction` entry.** No separate context snapshot. Reconstruction reuses Pi's battle-tested `buildSessionContext()`, so cold == warm context by construction. | -| D2 | How does cold start find where to resume? | A **resume-pointer marker**: on `session_compact`, append a tiny `custom`/`checkpoint` entry recording `resumeFromPosition` = the log position of the compaction's `firstKeptEntryId` (F4). Metadata only. This gives the existing `latestCheckpoint()` (F5) something to return. | -| D3 | Where does the loader live? | A new **additive** `SessionManager.openFromCheckpoint()` in pi-fork (F6). `openFromBackend` is left untouched for other callers. Falls back to `openFromBackend` when there is no marker. | -| D4 | Efficient tail read | `RedisSessionBackend` assigns each entry the stream id `"-0"` (positions are already monotonic 1-based via `INCR`), so `read(fromPosition)` becomes `XRANGE key -0 +` = O(tail). Backward-compatible for fresh sessions. | -| D5 | Cadence | **Ride Pi's compaction cadence.** No `everyK` policy (out of scope). | -| D6 | Budget metric | **Per-turn token-spend delta**: `getSessionStats().tokens.total − baseline` (baseline captured at turn start, so loaded-tail usage is excluded). We deliberately meter cumulative *spend*, **not** `getContextUsage()` context-window fill — the latter resets at compaction and is `null` right after (F8). A missing/`null` stat reading ⇒ **do not block** (defensive). | -| D7 | Budget cap source | Env `SH_BUDGET_TOKENS` (unset ⇒ voter disabled / inert). Optional `SH_BUDGET_MARGIN` for headroom. | -| D8 | Log entries written by the voter | Only an **`abort`** custom entry on block. **No** per-tool-call `vote` entry (the stale plan wrote one; that is log bloat — E5 only checks for `abort`). | -| D9 | Extension placement | Extensions live in **`harness/src/`** beside the existing `flush-extension.ts` — matching the real codebase. **No** new `packages/checkpoint` / `packages/budget-voter` (the stale plan's layout predates seeing that extensions live in the harness). Pure logic stays in small testable modules. | +| # | Decision | Choice | +| --- | ----------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| D1 | What is "the checkpoint"? | **Pi's native `compaction` entry.** No separate context snapshot. Reconstruction reuses Pi's battle-tested `buildSessionContext()`, so cold == warm context by construction. | +| D2 | How does cold start find where to resume? | A **resume-pointer marker**: on `session_compact`, append a tiny `custom`/`checkpoint` entry recording `resumeFromPosition` = the log position of the compaction's `firstKeptEntryId` (F4). Metadata only. This gives the existing `latestCheckpoint()` (F5) something to return. | +| D3 | Where does the loader live? | A new **additive** `SessionManager.openFromCheckpoint()` in pi-fork (F6). `openFromBackend` is left untouched for other callers. Falls back to `openFromBackend` when there is no marker. | +| D4 | Efficient tail read | `RedisSessionBackend` assigns each entry the stream id `"-0"` (positions are already monotonic 1-based via `INCR`), so `read(fromPosition)` becomes `XRANGE key -0 +` = O(tail). Backward-compatible for fresh sessions. | +| D5 | Cadence | **Ride Pi's compaction cadence.** No `everyK` policy (out of scope). | +| D6 | Budget metric | **Per-turn token-spend delta**: `getSessionStats().tokens.total − baseline` (baseline captured at turn start, so loaded-tail usage is excluded). We deliberately meter cumulative _spend_, **not** `getContextUsage()` context-window fill — the latter resets at compaction and is `null` right after (F8). A missing/`null` stat reading ⇒ **do not block** (defensive). | +| D7 | Budget cap source | Env `SH_BUDGET_TOKENS` (unset ⇒ voter disabled / inert). Optional `SH_BUDGET_MARGIN` for headroom. | +| D8 | Log entries written by the voter | Only an **`abort`** custom entry on block. **No** per-tool-call `vote` entry (the stale plan wrote one; that is log bloat — E5 only checks for `abort`). | +| D9 | Extension placement | Extensions live in **`harness/src/`** beside the existing `flush-extension.ts` — matching the real codebase. **No** new `packages/checkpoint` / `packages/budget-voter` (the stale plan's layout predates seeing that extensions live in the harness). Pure logic stays in small testable modules. | --- @@ -175,8 +175,8 @@ serverless-harness/ - **append**: replace `xAdd(key, "*", …)` with `xAdd(key, \`${position}-0\`, …)`. The position is already computed before the add. Entries remain readable identically. -- **read(fromPosition)**: `xRange(key, \`${fromPosition}-0\`, "+")` and drop the post-filter. - (Keep a defensive position filter only if mixing old `"*"`-id sessions — but those are +- **read(fromPosition)**: `xRange(key, \`${fromPosition}-0\`, "+")`and drop the post-filter. +(Keep a defensive position filter only if mixing old`"*"`-id sessions — but those are pre-M5 throwaway dev sessions; D4 scopes this to fresh sessions.) - **positionOfId(sid, id)**: read entries (O(N), acceptable — see §7) and return the `position` of the stored entry whose decoded `entry.id === id`, or `null`. @@ -214,11 +214,12 @@ checkpoint marker is a Pi `CustomEntry` (`type:"custom"`, `customType:"checkpoin ```ts export function checkpointExtension(store: LogStore, sm: SessionManager): ExtensionFactory { return (pi) => { - pi.on("session_compact", async (e) => { // e.compactionEntry (F1) + pi.on('session_compact', async (e) => { + // e.compactionEntry (F1) const sid = sm.getSessionId(); - const pos = await store.positionOfId(sid, e.compactionEntry.firstKeptEntryId); // F4 + const pos = await store.positionOfId(sid, e.compactionEntry.firstKeptEntryId); // F4 if (pos != null) { - sm.appendCustomEntry("checkpoint", { resumeFromPosition: pos }); // rides flush + sm.appendCustomEntry('checkpoint', { resumeFromPosition: pos }); // rides flush } }); }; @@ -232,32 +233,41 @@ flows through the existing buffered backend and flush path. Writing through the ### 4.5 Budget voter (`harness/src/budget-voter.ts`) ```ts -export interface BudgetState { spent: number; estimated: number; limit: number; } +export interface BudgetState { + spent: number; + estimated: number; + limit: number; +} export type BudgetDecision = - | { decision: "commit" } - | { decision: "abort"; reason: "budget_exceeded" }; + { decision: 'commit' } | { decision: 'abort'; reason: 'budget_exceeded' }; export function decideBudget(s: BudgetState): BudgetDecision { - if (!Number.isFinite(s.limit) || s.limit <= 0) return { decision: "commit" }; // disabled + if (!Number.isFinite(s.limit) || s.limit <= 0) return { decision: 'commit' }; // disabled return s.spent + s.estimated > s.limit - ? { decision: "abort", reason: "budget_exceeded" } - : { decision: "commit" }; + ? { decision: 'abort', reason: 'budget_exceeded' } + : { decision: 'commit' }; } -export function budgetVoterExtension(sm: SessionManager, opts: { - limit: number; margin?: number; -}): ExtensionFactory { +export function budgetVoterExtension( + sm: SessionManager, + opts: { + limit: number; + margin?: number; + }, +): ExtensionFactory { return (pi) => { let baseline: number | null = null; - pi.on("session_start", (_e, ctx) => { baseline = sessionStatsTotal(ctx); }); - pi.on("tool_call", (_e, ctx) => { - const total = sessionStatsTotal(ctx); // cumulative spend (D6); NOT getContextUsage (F8) - if (baseline == null || total == null) return {}; // null ⇒ don't block (D6) + pi.on('session_start', (_e, ctx) => { + baseline = sessionStatsTotal(ctx); + }); + pi.on('tool_call', (_e, ctx) => { + const total = sessionStatsTotal(ctx); // cumulative spend (D6); NOT getContextUsage (F8) + if (baseline == null || total == null) return {}; // null ⇒ don't block (D6) const spent = total - baseline; const d = decideBudget({ spent, estimated: opts.margin ?? 0, limit: opts.limit }); - if (d.decision === "abort") { - sm.appendCustomEntry("abort", { reason: d.reason, spent, limit: opts.limit }); // D8 - return { block: true, reason: "Session token budget exceeded" }; // F7 + if (d.decision === 'abort') { + sm.appendCustomEntry('abort', { reason: d.reason, spent, limit: opts.limit }); // D8 + return { block: true, reason: 'Session token budget exceeded' }; // F7 } return {}; }); @@ -278,10 +288,12 @@ export function budgetVoterExtension(sm: SessionManager, opts: { ## 5. Verification gate ### Unit + - `decideBudget`: commits below cap; aborts when `spent + estimated > limit`; commits when limit ≤ 0 (disabled). - `RedisSessionBackend`: `read(fromPosition)` returns exactly the tail (positions ≥ fromPosition) using stream-id seek; `positionOfId` returns the right position and `null` for an unknown id. ### Integration (real Redis on localhost:6379) + - **Reconstruction parity (the gate):** drive a session that compacts at least once (append a real `compaction` entry + a `checkpoint` marker), then assert `buildSessionContext()` from `openFromCheckpoint` deep-equals the result from `openFromBackend`. - **Resume recalls kept context:** after a compaction, a fresh `openFromCheckpoint` includes the compaction summary and every kept-tail message; the slice read is strictly smaller than the full log. @@ -289,6 +301,7 @@ export function budgetVoterExtension(sm: SessionManager, opts: { - **Budget voter:** with a low `SH_BUDGET_TOKENS`, a synthetic over-baseline `getSessionStats` causes the `tool_call` handler to return `{ block:true }` and append exactly one `abort` entry; with the cap unset, no block and no `abort` entry. ### Build/regression + - `pnpm -C pi-fork/packages/coding-agent build` and its test suite pass after `openFromCheckpoint` (analyze logs via subagent per the context-budget rule). - Existing harness, session-backend, k8s-sandbox suites stay green. @@ -296,27 +309,27 @@ export function budgetVoterExtension(sm: SessionManager, opts: { ## 6. Deviations from the stale plan (Tasks 13–14) -| Plan (Task 13/14) | This spec | Why | -|---|---|---| -| `packages/checkpoint` + `packages/budget-voter` | Extensions in `harness/src/` | Matches the real codebase (`flush-extension.ts` lives here). | -| `shouldCheckpoint` + `everyK` cadence | Ride Pi's compaction cadence | Pi compaction already is the checkpoint; `everyK` is out of scope (deferred to E2 need). | -| `writeCheckpoint(context)` snapshotting context | Tiny resume-pointer marker | Context lives in Pi's `compaction` entry; don't duplicate it. | +| Plan (Task 13/14) | This spec | Why | +| -------------------------------------------------------------------------- | ------------------------------------------------------------- | -------------------------------------------------------------------------------------------- | +| `packages/checkpoint` + `packages/budget-voter` | Extensions in `harness/src/` | Matches the real codebase (`flush-extension.ts` lives here). | +| `shouldCheckpoint` + `everyK` cadence | Ride Pi's compaction cadence | Pi compaction already is the checkpoint; `everyK` is out of scope (deferred to E2 need). | +| `writeCheckpoint(context)` snapshotting context | Tiny resume-pointer marker | Context lives in Pi's `compaction` entry; don't duplicate it. | | `decideBudget({cumulative, estimated, limit})` over an external cumulative | `decideBudget({spent, estimated, limit})` over per-turn delta | The serverless model is per-turn; cross-turn cumulative needs a persisted ledger (deferred). | -| `vote` entry per allowed tool call | Only `abort` on block | Avoids log bloat; E5 only inspects `abort`. | -| `FileSessionBackend`, `CtxMsg`, `reconstruct.ts` | — (do not exist) | Re-targeted to the real `LogStore`/`SessionStorageBackend` + Pi's `buildSessionContext`. | +| `vote` entry per allowed tool call | Only `abort` on block | Avoids log bloat; E5 only inspects `abort`. | +| `FileSessionBackend`, `CtxMsg`, `reconstruct.ts` | — (do not exist) | Re-targeted to the real `LogStore`/`SessionStorageBackend` + Pi's `buildSessionContext`. | --- ## 7. Residual risks -| Risk | Impact | Mitigation | -|------|--------|------------| -| `positionOfId` scans the whole stream (O(N)). | Cost at marker-write time. | Runs only at `session_compact` (rare, already LLM-costly). Acceptable; revisit if compaction frequency rises. | -| Stream-id scheme change (`"*"` → `"-0"`) is incompatible with sessions written by older code. | Old dev sessions may misread. | Scoped to fresh sessions (D4); pre-M5 sessions are throwaway. Document in README. | -| `getSessionStats` delta is a "recent spend" proxy, not exact per-turn spend (loaded-tail messages carry prior usage; baseline subtraction mitigates but compaction within a turn can perturb it). | Voter trips slightly early/late. | Acceptable for PoC + E5 (single expensive task). `null`-handling prevents false blocks. Documented. | -| The loader's benefit is local (I/O + walk), not LLM latency — see §1. | E2 measured the wrong thing → "no effect". | Spec states E2 must measure **local reconstruction cost**; flagged for the experiments milestone. | -| Tail-load loses access to pre-compaction entries by id (branches/labels). | Non-issue for serverless resume (continues the leaf). | `openFromBackend` remains available for any full-history use; loader is opt-in. | -| `openFromCheckpoint` tail-load excludes the session-start `thinking_level_change` entry (pre-`firstKeptEntryId`), so `buildSessionContext().thinkingLevel` resets to `"off"` on the first post-compaction resume. | `thinkingLevel` is wrong for one turn. | Self-heals on next turn (Pi re-appends the entry). Safe while thinking level is settings-governed. `model` is unaffected (re-set by every assistant message, always in the kept tail). Locked by a dedicated test. | +| Risk | Impact | Mitigation | +| ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | +| `positionOfId` scans the whole stream (O(N)). | Cost at marker-write time. | Runs only at `session_compact` (rare, already LLM-costly). Acceptable; revisit if compaction frequency rises. | +| Stream-id scheme change (`"*"` → `"-0"`) is incompatible with sessions written by older code. | Old dev sessions may misread. | Scoped to fresh sessions (D4); pre-M5 sessions are throwaway. Document in README. | +| `getSessionStats` delta is a "recent spend" proxy, not exact per-turn spend (loaded-tail messages carry prior usage; baseline subtraction mitigates but compaction within a turn can perturb it). | Voter trips slightly early/late. | Acceptable for PoC + E5 (single expensive task). `null`-handling prevents false blocks. Documented. | +| The loader's benefit is local (I/O + walk), not LLM latency — see §1. | E2 measured the wrong thing → "no effect". | Spec states E2 must measure **local reconstruction cost**; flagged for the experiments milestone. | +| Tail-load loses access to pre-compaction entries by id (branches/labels). | Non-issue for serverless resume (continues the leaf). | `openFromBackend` remains available for any full-history use; loader is opt-in. | +| `openFromCheckpoint` tail-load excludes the session-start `thinking_level_change` entry (pre-`firstKeptEntryId`), so `buildSessionContext().thinkingLevel` resets to `"off"` on the first post-compaction resume. | `thinkingLevel` is wrong for one turn. | Self-heals on next turn (Pi re-appends the entry). Safe while thinking level is settings-governed. `model` is unaffected (re-set by every assistant message, always in the kept tail). Locked by a dedicated test. | --- @@ -331,4 +344,4 @@ bounds LLM-facing context. --- -*Assisted-By: Claude (Anthropic AI) * +_Assisted-By: Claude (Anthropic AI) _ diff --git a/docs/specs/2026-06-24-m6-experiments-design.md b/docs/specs/2026-06-24-m6-experiments-design.md index a3b3384..fa4e431 100644 --- a/docs/specs/2026-06-24-m6-experiments-design.md +++ b/docs/specs/2026-06-24-m6-experiments-design.md @@ -51,12 +51,12 @@ M6 is **done** when: ### 1.1 Honest framing (inherited from M5 §1/§8) -The pi-track plan's E2 measured *end-to-end cold-start latency* (message → first token) under an +The pi-track plan's E2 measured _end-to-end cold-start latency_ (message → first token) under an `everyK` checkpoint cadence. M5 §1 established this is the wrong instrument: Pi's native compaction already bounds the LLM context (`buildSessionContext()` assembles only `[summary, …kept tail, …post-compaction]`), so end-to-end latency is dominated by an already-bounded LLM call and would show **"no effect."** The loader's real, measurable win is -**local**: the Redis read, `_buildIndex`, and the leaf→root walk. M6/E2 measures *that*, directly +**local**: the Redis read, `_buildIndex`, and the leaf→root walk. M6/E2 measures _that_, directly and deterministically, without an LLM. The `everyK` knob does not exist (we ride Pi's native compaction cadence — M5 D5), so it plays no part here. @@ -76,38 +76,38 @@ compaction cadence — M5 D5), so it plays no part here. - **`everyK` / forced-checkpoint cadence** (deferred in M5; ride native compaction). - **Cross-turn / cumulative budget ledger** (deferred in M5). - **Knative/HTTP-driven experiments and a Kind cluster.** M6 is in-process + local Redis only. -- **E1, E3, E4** — these are end-to-end *cluster* experiments; see §8.1. +- **E1, E3, E4** — these are end-to-end _cluster_ experiments; see §8.1. - **The pi-track plan's Python `experiments/` drivers** — re-targeted to in-process TS. --- ## 2. Discovery findings (the design rests on these) -| # | Finding | Evidence | -|---|---------|----------| -| F1 | M5 is merged to `main`: `openFromCheckpoint` + `markerResumePosition` (pi-fork), `RedisSessionBackend` deterministic stream ids + `positionOfId`, `checkpoint-extension.ts`, `budget-voter.ts`, wired in `run-turn.ts`. | `8083a27` (PR #2); `run-turn.ts:48` calls `openFromCheckpoint`. | -| F2 | Both loaders accept an injected `backend: SessionStorageBackend` and call `backend.read(...)`. `openFromBackend` reads **all** entries; `openFromCheckpoint` reads only `marker.resumeFromPosition`-forward. ⇒ wrapping the injected backend captures the read-volume difference with no fork change. | `session-manager.ts:1430` (`openFromBackend`), `:1457` (`openFromCheckpoint`). | -| F3 | `SessionStorageBackend` is a 4-method interface: `append`, `read(sid, fromPosition?)`, `latestCheckpoint(sid)`, `list()`. A decorator is small. | `core/session-storage-backend.ts:14–22`. | -| F4 | The M5 `checkpoint.test.ts` builds a real compacted session in Redis (append entries → `appendCompaction` → checkpoint marker) and asserts `openFromCheckpoint` parity vs `openFromBackend`. This is the template for the E2 synthetic fixture. | `harness/test/checkpoint.test.ts` (M5). | -| F5 | The budget voter meters per-turn spend = `sessionSpendTotal(ctx) − baseline` (baseline captured at `session_start`); on `tool_call` it blocks and appends one `abort` via `sm.appendCustomEntry("abort", …)`. `sessionSpendTotal` reads `ctx.sessionManager.getBranch()` assistant `usage`. Returns `0` for an empty branch, `null` only when the branch is unavailable (a missing/`null` reading ⇒ do not block). | `harness/src/budget-voter.ts` (M5). | -| F6 | The production turn path hardcodes the model: `getModel("anthropic", "claude-opus-4-8")`. Credentials already flow via `ANTHROPIC_AUTH_TOKEN` / `ANTHROPIC_BASE_URL` (+ optional `anthropicAuthToken`/`anthropicBaseUrl` on `TurnConfig`); the gateway path overrides `baseUrl` and injects `Authorization: Bearer` while nulling `x-api-key`. | `run-turn.ts:78`, `:39–42`, `:79–95`. | -| F7 | `pnpm-workspace.yaml` declares `packages/*` and `harness`. A new top-level `experiments` dir must be added to the workspace globs. | `pnpm-workspace.yaml`. | +| # | Finding | Evidence | +| --- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ------------------------------------------------------------------------------ | +| F1 | M5 is merged to `main`: `openFromCheckpoint` + `markerResumePosition` (pi-fork), `RedisSessionBackend` deterministic stream ids + `positionOfId`, `checkpoint-extension.ts`, `budget-voter.ts`, wired in `run-turn.ts`. | `8083a27` (PR #2); `run-turn.ts:48` calls `openFromCheckpoint`. | +| F2 | Both loaders accept an injected `backend: SessionStorageBackend` and call `backend.read(...)`. `openFromBackend` reads **all** entries; `openFromCheckpoint` reads only `marker.resumeFromPosition`-forward. ⇒ wrapping the injected backend captures the read-volume difference with no fork change. | `session-manager.ts:1430` (`openFromBackend`), `:1457` (`openFromCheckpoint`). | +| F3 | `SessionStorageBackend` is a 4-method interface: `append`, `read(sid, fromPosition?)`, `latestCheckpoint(sid)`, `list()`. A decorator is small. | `core/session-storage-backend.ts:14–22`. | +| F4 | The M5 `checkpoint.test.ts` builds a real compacted session in Redis (append entries → `appendCompaction` → checkpoint marker) and asserts `openFromCheckpoint` parity vs `openFromBackend`. This is the template for the E2 synthetic fixture. | `harness/test/checkpoint.test.ts` (M5). | +| F5 | The budget voter meters per-turn spend = `sessionSpendTotal(ctx) − baseline` (baseline captured at `session_start`); on `tool_call` it blocks and appends one `abort` via `sm.appendCustomEntry("abort", …)`. `sessionSpendTotal` reads `ctx.sessionManager.getBranch()` assistant `usage`. Returns `0` for an empty branch, `null` only when the branch is unavailable (a missing/`null` reading ⇒ do not block). | `harness/src/budget-voter.ts` (M5). | +| F6 | The production turn path hardcodes the model: `getModel("anthropic", "claude-opus-4-8")`. Credentials already flow via `ANTHROPIC_AUTH_TOKEN` / `ANTHROPIC_BASE_URL` (+ optional `anthropicAuthToken`/`anthropicBaseUrl` on `TurnConfig`); the gateway path overrides `baseUrl` and injects `Authorization: Bearer` while nulling `x-api-key`. | `run-turn.ts:78`, `:39–42`, `:79–95`. | +| F7 | `pnpm-workspace.yaml` declares `packages/*` and `harness`. A new top-level `experiments` dir must be added to the workspace globs. | `pnpm-workspace.yaml`. | --- ## 3. Key decisions -| # | Decision | Choice | -|---|----------|--------| -| D1 | E2 session generation | **Synthetic, no LLM.** Programmatically append realistic `FileEntry` logs + a real `compaction` entry + a `checkpoint` marker to real Redis (per F4). Deterministic, scales to any N cheaply, reproducible, needs no key. A live model is needed only by E5. | -| D2 | E2 headline metric | **Entries read + bytes read** from Redis during reconstruction, via a counting backend decorator (no pi-fork change). #entries loaded is the common driver of the Redis read **and** `_buildIndex` **and** the leaf→root walk, so it faithfully proxies all three local costs. Wall-clock reconstruction time is a **secondary, illustrative** column. | -| D3 | E2 pass criterion | The `backend / checkpoint` read ratio **strictly increases with N** (checkpoint near-constant, backend linear). Concretely: assert the ratio at N=5000 is materially greater than at N=50 (and monotonic non-decreasing across the N series, within a small tolerance). Plus a `buildSessionContext()` parity re-confirmation. | -| D4 | E5 structural gate | **No key, real Redis.** Inject a synthetic over-cap spend at the voter/extension boundary; assert tool_call blocked **and** exactly one `abort` entry actually persisted in the Redis log; assert inert (no block, no `abort`) when `SH_BUDGET_TOKENS` unset. This is the M6 pass gate for E5. | -| D5 | E5 live run | **Key-gated, tiny cap.** `SH_BUDGET_TOKENS=1` + a prompt that forces a tool call ⇒ the first post-baseline `tool_call` deterministically trips the cap. Assert block + exactly one `abort` in Redis. Skips cleanly when `SH_RUN_LIVE`/key absent. | -| D6 | Experiment placement | **New `experiments/` pnpm workspace**, vitest, in-process. Mirrors the parent plan's `experiments/` intent in TypeScript. E2 + E5-structural always run (no key); E5-live is gated. | -| D7 | Model as runtime input | Env + config, defaults preserved: `getModel(provider, modelId)` where `provider = config?.provider ?? SH_MODEL_PROVIDER ?? "anthropic"` and `modelId = config?.model ?? SH_MODEL ?? "claude-opus-4-8"`. `TurnConfig` gains optional `model?` / `provider?`. No secrets in repo. | -| D8 | Provider scope | `SH_MODEL_PROVIDER` is **forward-looking** and exercised here only as `anthropic` (directly or via the litellm gateway through `ANTHROPIC_BASE_URL`). The auth-header injection (F6) is anthropic/gateway-shaped; a genuinely different provider would need its own key handling — documented limitation, not implemented. | -| D9 | Milestone scope | **One milestone, one branch** (`feat/m6-experiments`). E2 and E5 are independent experiments but small and share the workspace + model/credential config + real-Redis setup + reporting. No pi-fork edits ⇒ **no submodule branch**. | +| # | Decision | Choice | +| --- | ---------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | +| D1 | E2 session generation | **Synthetic, no LLM.** Programmatically append realistic `FileEntry` logs + a real `compaction` entry + a `checkpoint` marker to real Redis (per F4). Deterministic, scales to any N cheaply, reproducible, needs no key. A live model is needed only by E5. | +| D2 | E2 headline metric | **Entries read + bytes read** from Redis during reconstruction, via a counting backend decorator (no pi-fork change). #entries loaded is the common driver of the Redis read **and** `_buildIndex` **and** the leaf→root walk, so it faithfully proxies all three local costs. Wall-clock reconstruction time is a **secondary, illustrative** column. | +| D3 | E2 pass criterion | The `backend / checkpoint` read ratio **strictly increases with N** (checkpoint near-constant, backend linear). Concretely: assert the ratio at N=5000 is materially greater than at N=50 (and monotonic non-decreasing across the N series, within a small tolerance). Plus a `buildSessionContext()` parity re-confirmation. | +| D4 | E5 structural gate | **No key, real Redis.** Inject a synthetic over-cap spend at the voter/extension boundary; assert tool_call blocked **and** exactly one `abort` entry actually persisted in the Redis log; assert inert (no block, no `abort`) when `SH_BUDGET_TOKENS` unset. This is the M6 pass gate for E5. | +| D5 | E5 live run | **Key-gated, tiny cap.** `SH_BUDGET_TOKENS=1` + a prompt that forces a tool call ⇒ the first post-baseline `tool_call` deterministically trips the cap. Assert block + exactly one `abort` in Redis. Skips cleanly when `SH_RUN_LIVE`/key absent. | +| D6 | Experiment placement | **New `experiments/` pnpm workspace**, vitest, in-process. Mirrors the parent plan's `experiments/` intent in TypeScript. E2 + E5-structural always run (no key); E5-live is gated. | +| D7 | Model as runtime input | Env + config, defaults preserved: `getModel(provider, modelId)` where `provider = config?.provider ?? SH_MODEL_PROVIDER ?? "anthropic"` and `modelId = config?.model ?? SH_MODEL ?? "claude-opus-4-8"`. `TurnConfig` gains optional `model?` / `provider?`. No secrets in repo. | +| D8 | Provider scope | `SH_MODEL_PROVIDER` is **forward-looking** and exercised here only as `anthropic` (directly or via the litellm gateway through `ANTHROPIC_BASE_URL`). The auth-header injection (F6) is anthropic/gateway-shaped; a genuinely different provider would need its own key handling — documented limitation, not implemented. | +| D9 | Milestone scope | **One milestone, one branch** (`feat/m6-experiments`). E2 and E5 are independent experiments but small and share the workspace + model/credential config + real-Redis setup + reporting. No pi-fork edits ⇒ **no submodule branch**. | --- @@ -171,13 +171,13 @@ export interface TurnConfig { cwd?: string; anthropicBaseUrl?: string; anthropicAuthToken?: string; - model?: string; // NEW - provider?: string; // NEW + model?: string; // NEW + provider?: string; // NEW } // replaces run-turn.ts:78 -const provider = config?.provider ?? process.env.SH_MODEL_PROVIDER ?? "anthropic"; -const modelId = config?.model ?? process.env.SH_MODEL ?? "claude-opus-4-8"; +const provider = config?.provider ?? process.env.SH_MODEL_PROVIDER ?? 'anthropic'; +const modelId = config?.model ?? process.env.SH_MODEL ?? 'claude-opus-4-8'; const baseModel = getModel(provider, modelId); ``` @@ -229,10 +229,10 @@ pos m+1.. post-compaction messages Two facts make a simple "read from the compaction entry forward" insufficient, and motivate the separate `checkpoint` marker: -1. **The kept tail sits *before* the compaction entry (F4).** Pi appends the compaction entry - *after* the messages it retains, so reading forward from the compaction entry's position would +1. **The kept tail sits _before_ the compaction entry (F4).** Pi appends the compaction entry + _after_ the messages it retains, so reading forward from the compaction entry's position would drop the kept tail (positions `k..m-1`) and reconstruct the wrong context. The correct resume - point is `firstKeptEntryId` at position `k`, which is *earlier* in the log. + point is `firstKeptEntryId` at position `k`, which is _earlier_ in the log. 2. **Redis Streams seek by position, not by content-id.** The compaction entry holds `firstKeptEntryId` as a string id buried in its payload. Translating that id → stream position requires reading entries until it is found — an O(N) scan (`positionOfId`), which is exactly the @@ -278,44 +278,48 @@ so the model has something to call. ## 5. Verification gate ### Unit (no key) + - `CountingBackend`: read tallies match expected entries and byte totals; `reset()` zeroes; non-`read` methods delegate. - `session-fixture`: produced session has N+compaction+marker; `latestCheckpoint()` returns the marker; `openFromCheckpoint` reads strictly fewer entries than `openFromBackend`. ### Integration — real Redis on `localhost:6379` (no key) + - **E2 ratio gate (primary):** across N ∈ {50,200,1000,5000}, checkpoint entries/bytes ≈ constant; backend grows ~linearly; `ratio(N=5000) ≫ ratio(N=50)` and non-decreasing across the series (tolerance documented). `buildSessionContext()` parity holds at every N. - **E5 structural gate (primary):** over-cap spend ⇒ `{ block:true }` + exactly one persisted `abort`; unset cap ⇒ no block, no `abort`. ### Live (manual, key-gated) + - **E5 live:** tiny-cap real-model run blocks and records exactly one `abort`. Skips with no key. ### Build/regression + - `pnpm -C experiments test` green; existing harness / session-backend / k8s-sandbox suites green; pi-fork untouched (no submodule change). Analyze long logs via subagents (context-budget rule). --- ## 6. Deviations from the pi-track plan (Tasks 18, 21) -| Plan (Task 18/21) | This spec | Why | -|---|---|---| -| E2 = end-to-end cold-start **latency** (message→first-token) | E2 = **local reconstruction cost** (entries/bytes read) | Pi compaction already bounds LLM context; latency would show "no effect" (M5 §1, §1.1 here). | -| `everyK` cadence on/off | Ride Pi's native compaction; no `everyK` | The knob doesn't exist (M5 D5); fixture compacts once. | -| Python drivers, HTTP against deployed Knative service, `kubectl scale` | In-process TypeScript vitest, local Redis | E2 measures in-process Redis reads + the `buildSessionContext` walk — not observable over HTTP. E5 drives the voter directly. | -| E5: realistic 50k cap + "refactor everything", assert elapsed | E5: no-key structural gate (synthetic spend) + key-gated **tiny-cap** live run | Deterministic CI gate without a key; tiny cap makes the live breach deterministic. | -| `redis-cli XRANGE … | count "abort"` via `kubectl exec` | Assert exactly one `abort` in Redis from the test process | No cluster; direct Redis assertion is exact. | +| Plan (Task 18/21) | This spec | Why | +| ---------------------------------------------------------------------- | ------------------------------------------------------------------------------ | ----------------------------------------------------------------------------------------------------------------------------- | +| E2 = end-to-end cold-start **latency** (message→first-token) | E2 = **local reconstruction cost** (entries/bytes read) | Pi compaction already bounds LLM context; latency would show "no effect" (M5 §1, §1.1 here). | +| `everyK` cadence on/off | Ride Pi's native compaction; no `everyK` | The knob doesn't exist (M5 D5); fixture compacts once. | +| Python drivers, HTTP against deployed Knative service, `kubectl scale` | In-process TypeScript vitest, local Redis | E2 measures in-process Redis reads + the `buildSessionContext` walk — not observable over HTTP. E5 drives the voter directly. | +| E5: realistic 50k cap + "refactor everything", assert elapsed | E5: no-key structural gate (synthetic spend) + key-gated **tiny-cap** live run | Deterministic CI gate without a key; tiny cap makes the live breach deterministic. | +| `redis-cli XRANGE … | count "abort"`via`kubectl exec` | Assert exactly one `abort` in Redis from the test process | No cluster; direct Redis assertion is exact. | --- ## 7. Residual risks -| Risk | Impact | Mitigation | -|------|--------|------------| -| Counting at the `SessionStorageBackend` boundary measures entries/bytes returned, not raw Redis wire bytes. | Slight abstraction from true I/O. | Faithful proxy for the O(tail) vs O(total) story (the quantity that differs); documented. Wall-clock column gives a tangible secondary read. | -| Wall-clock reconstruction time is noisy on a dev box. | Secondary column varies run-to-run. | Reported as illustrative only; the gate is the deterministic entries/bytes ratio (D3). | -| Live E5 needs a tool the model will actually call without a sandbox pod. | Live breach may not fire. | Resolved in the plan (§4.7): set `KAGENTI_SANDBOX_POD` or register a trivial tool. Live run is **not** the pass gate (D4 structural gate is). | -| Synthetic fixture diverges from genuinely-compacted sessions. | E2 measures an unrealistic shape. | Built through the same `RedisSessionBackend`/`appendCompaction` path as `checkpoint.test.ts` (F4); a unit test asserts `latestCheckpoint`/parity hold on it. | -| `SH_MODEL_PROVIDER` ≠ anthropic is untested (D8). | False expectation of multi-provider support. | Documented as forward-looking; default and tests use anthropic (incl. gateway). | -| New workspace glob / cross-package deps misconfigured. | `pnpm -C experiments test` won't resolve `harness`/pi-fork. | Mirror the existing `harness` package's workspace deps; build-order gotchas from M2/M4 noted in the plan. | -| The harness never calls `bindExtensions()`, so `session_start` is **not** emitted in the headless `runTurn` path (only interactive/print/rpc modes emit it). | The voter's original `session_start`-captured baseline stayed `null`, so it never blocked against a real model — found by the live E5 run; the structural/unit tests masked it by firing `session_start` by hand. | **Fixed:** the voter no longer depends on `session_start`; `run-turn` computes the pre-turn baseline (`branchSpend(sessionManager)`) and injects it. **Open for M7:** verify whether `session_compact` (checkpoint marker) and `turn_end` (flush) *also* require `bindExtensions` — if so, the checkpoint marker is never written in production and `openFromCheckpoint` always falls back to full replay, which M7's E3-mobility depends on. | +| Risk | Impact | Mitigation | +| ------------------------------------------------------------------------------------------------------------------------------------------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| Counting at the `SessionStorageBackend` boundary measures entries/bytes returned, not raw Redis wire bytes. | Slight abstraction from true I/O. | Faithful proxy for the O(tail) vs O(total) story (the quantity that differs); documented. Wall-clock column gives a tangible secondary read. | +| Wall-clock reconstruction time is noisy on a dev box. | Secondary column varies run-to-run. | Reported as illustrative only; the gate is the deterministic entries/bytes ratio (D3). | +| Live E5 needs a tool the model will actually call without a sandbox pod. | Live breach may not fire. | Resolved in the plan (§4.7): set `KAGENTI_SANDBOX_POD` or register a trivial tool. Live run is **not** the pass gate (D4 structural gate is). | +| Synthetic fixture diverges from genuinely-compacted sessions. | E2 measures an unrealistic shape. | Built through the same `RedisSessionBackend`/`appendCompaction` path as `checkpoint.test.ts` (F4); a unit test asserts `latestCheckpoint`/parity hold on it. | +| `SH_MODEL_PROVIDER` ≠ anthropic is untested (D8). | False expectation of multi-provider support. | Documented as forward-looking; default and tests use anthropic (incl. gateway). | +| New workspace glob / cross-package deps misconfigured. | `pnpm -C experiments test` won't resolve `harness`/pi-fork. | Mirror the existing `harness` package's workspace deps; build-order gotchas from M2/M4 noted in the plan. | +| The harness never calls `bindExtensions()`, so `session_start` is **not** emitted in the headless `runTurn` path (only interactive/print/rpc modes emit it). | The voter's original `session_start`-captured baseline stayed `null`, so it never blocked against a real model — found by the live E5 run; the structural/unit tests masked it by firing `session_start` by hand. | **Fixed:** the voter no longer depends on `session_start`; `run-turn` computes the pre-turn baseline (`branchSpend(sessionManager)`) and injects it. **Open for M7:** verify whether `session_compact` (checkpoint marker) and `turn_end` (flush) _also_ require `bindExtensions` — if so, the checkpoint marker is never written in production and `openFromCheckpoint` always falls back to full replay, which M7's E3-mobility depends on. | --- @@ -376,4 +380,4 @@ No secrets in the repo; model + provider + key are env-only. --- -*Assisted-By: Claude (Anthropic AI) * +_Assisted-By: Claude (Anthropic AI) _ diff --git a/docs/specs/2026-06-25-m7-cluster-experiments-design.md b/docs/specs/2026-06-25-m7-cluster-experiments-design.md index ac3c106..661c2b3 100644 --- a/docs/specs/2026-06-25-m7-cluster-experiments-design.md +++ b/docs/specs/2026-06-25-m7-cluster-experiments-design.md @@ -10,8 +10,8 @@ extending `deploy/knative/smoke.sh`. Parent plan: [Serverless Harness (Pi Track) Implementation Plan](../../../docs/research/2026-06-10-serverless-harness-pi-track-plan.md) — experiments **E1 (Task 17)**, **E3 (Task 19)**, **E4 (Task 20)**. Predecessors: [M4 — Knative Serverless Wrapper](2026-06-17-m4-knative-serverless-wrapper-design.md) (the deploy this builds on) and [M6 — Experiments E2 + E5](2026-06-24-m6-experiments-design.md) (§8.1 defined this sequel). -> **Milestone numbering.** Repo milestone **M7**. M6 (E2/E5) measured *local* behavior in-process; -> M7 measures the *deployed serverless system* end-to-end. The parent plan's literal drivers +> **Milestone numbering.** Repo milestone **M7**. M6 (E2/E5) measured _local_ behavior in-process; +> M7 measures the _deployed serverless system_ end-to-end. The parent plan's literal drivers > (Python, `requests`, ksvc `pi-harness` in `kagenti-system`) do **not** apply — re-targeted to > bash + the real deploy (ksvc `serverless-harness` in namespace `default`). @@ -29,13 +29,14 @@ M7 is **done** when, on the running Kind cluster, three idempotent bash drivers 1. **E1 — economics:** serverless (`min-scale=0`) pod-seconds are materially below persistent (`min-scale=1`) over an idle-heavy workload. -2. **E3 — mobility:** after the session's pod is gone, a *fresh* instance answers a follow-up that +2. **E3 — mobility:** after the session's pod is gone, a _fresh_ instance answers a follow-up that requires earlier-turn context, reconstructed from the Redis log. 3. **E4 — recovery:** force-killing the pod mid-session loses zero completed turns; the next turn continues from the persisted log. 4. `smoke.sh` still passes after its choreography is factored into a shared lib; pi-fork untouched. ### In scope + - A reusable bash lib (`deploy/knative/lib.sh`) factored out of `smoke.sh`. - Three drivers `deploy/knative/e1-economics.sh`, `e3-mobility.sh`, `e4-recovery.sh`. - A setup/harden step: correct target names, (re)create the `llm-credentials` secret from env, set @@ -43,6 +44,7 @@ M7 is **done** when, on the running Kind cluster, three idempotent bash drivers - `deploy/knative/EXPERIMENTS.md` recording the runs. ### Out of scope + - Exact cloud billing model (pod-seconds is the cost proxy); production autoscaler tuning. - Non-Knative platforms; multi-node scale. - The `session_shutdown` dead-handler cleanup in `flush-extension.ts` (noted in §7; harmless). @@ -52,29 +54,29 @@ M7 is **done** when, on the running Kind cluster, three idempotent bash drivers ## 2. Discovery findings (the design rests on these) -| # | Finding | Evidence | -|---|---------|----------| -| F1 | The harness extension lifecycle events M7 depends on **fire in the headless `runTurn` path**: `session_compact` (→ checkpoint marker) at `agent-session.ts:1733` (manual) / `:2013` (auto-compaction), and `turn_end` (→ flush) at `agent-loop.ts:218`; both via `_extensionRunner.emit`, no `bindExtensions` gate. ⇒ the checkpoint marker **is** written in production; the M5/M6 fast path is not dormant; E3-mobility's premise holds. | Subagent trace, 2026-06-25. | -| F2 | `session_shutdown` is **never** emitted in headless (only `reload()` / interactive `AgentSessionRuntime`), so `flush-extension.ts:15`'s handler is dead — but harmless, since `turn_end` fires each turn and `run-turn.ts` calls an explicit final `backend.flush()`. | Same trace; `flush-extension.ts:14-15`. | -| F3 | The deploy uses ksvc **`serverless-harness`** in namespace **`default`** (not the parent plan's `pi-harness`/`kagenti-system`). Annotations: `min-scale:0`, `max-scale:5`, `scale-to-zero-pod-retention-period:30s`, `target-burst-capacity:0`, `containerConcurrency:1`. | `deploy/knative/service.yaml:4-16`. | -| F4 | LLM creds already reach the ksvc via the `llm-credentials` secret (`ANTHROPIC_API_KEY` / `ANTHROPIC_BASE_URL` / `ANTHROPIC_AUTH_TOKEN`). The ksvc does **not** set `SH_MODEL` (defaults to `claude-opus-4-8`). M4's smoke did real gateway turns. | `service.yaml:23-44`; `SMOKE.md`. | -| F5 | `smoke.sh` already implements the core choreography: POST `/turn` (create + resume), wait-for-scale-to-zero, cold-start resume assertion. E3/E4 are extensions of it. | `deploy/knative/smoke.sh`, `SMOKE.md` Claims 2–5. | -| F6 | The Kind cluster `sh-knative` is running, so M7 can execute against the live deploy. | env. | +| # | Finding | Evidence | +| --- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ------------------------------------------------- | +| F1 | The harness extension lifecycle events M7 depends on **fire in the headless `runTurn` path**: `session_compact` (→ checkpoint marker) at `agent-session.ts:1733` (manual) / `:2013` (auto-compaction), and `turn_end` (→ flush) at `agent-loop.ts:218`; both via `_extensionRunner.emit`, no `bindExtensions` gate. ⇒ the checkpoint marker **is** written in production; the M5/M6 fast path is not dormant; E3-mobility's premise holds. | Subagent trace, 2026-06-25. | +| F2 | `session_shutdown` is **never** emitted in headless (only `reload()` / interactive `AgentSessionRuntime`), so `flush-extension.ts:15`'s handler is dead — but harmless, since `turn_end` fires each turn and `run-turn.ts` calls an explicit final `backend.flush()`. | Same trace; `flush-extension.ts:14-15`. | +| F3 | The deploy uses ksvc **`serverless-harness`** in namespace **`default`** (not the parent plan's `pi-harness`/`kagenti-system`). Annotations: `min-scale:0`, `max-scale:5`, `scale-to-zero-pod-retention-period:30s`, `target-burst-capacity:0`, `containerConcurrency:1`. | `deploy/knative/service.yaml:4-16`. | +| F4 | LLM creds already reach the ksvc via the `llm-credentials` secret (`ANTHROPIC_API_KEY` / `ANTHROPIC_BASE_URL` / `ANTHROPIC_AUTH_TOKEN`). The ksvc does **not** set `SH_MODEL` (defaults to `claude-opus-4-8`). M4's smoke did real gateway turns. | `service.yaml:23-44`; `SMOKE.md`. | +| F5 | `smoke.sh` already implements the core choreography: POST `/turn` (create + resume), wait-for-scale-to-zero, cold-start resume assertion. E3/E4 are extensions of it. | `deploy/knative/smoke.sh`, `SMOKE.md` Claims 2–5. | +| F6 | The Kind cluster `sh-knative` is running, so M7 can execute against the live deploy. | env. | --- ## 3. Key decisions -| # | Decision | Choice | -|---|----------|--------| -| D1 | Driver style | **Bash, extending `smoke.sh`** (kubectl + curl + jq), under `deploy/knative/`. Cluster ops are native to kubectl/curl; `smoke.sh` already does ~80% of the choreography; no new toolchain. | -| D2 | E1 cost metric | **Sampled pod-seconds**: poll `kubectl get pods -l serving.knative.dev/service=serverless-harness -n default` every `SAMPLE_INTERVAL` (default 5s); pod-seconds = Σ(running harness pods × interval). Run the same workload once at `min-scale=1`, once at `min-scale=0`. | -| D3 | E1 pass criterion | **serverless ≤ 0.6 × persistent** pod-seconds (≥40% reduction) over the idle-heavy pattern, AND report both absolute numbers + ratio. Magnitude depends on the idle/retention ratio (documented); the gate is directional + a clear margin. | -| D4 | Scope | **One milestone**, one branch, shared setup + `lib.sh` + one `EXPERIMENTS.md`. | -| D5 | Experiment model | Set **`SH_MODEL=claude-haiku-4-5`** (dash-form, anthropic) on the ksvc — cheap/fast for the many real turns. The M6 `requireModel` guard protects against a bad id. **Note:** non-anthropic models (e.g. Gemini) through the litellm gateway are **not** a config-only switch — the gateway bridge keeps the model's pi-ai `api`, and `gemini-2.0-flash` is registered under `google` with the native `google-generative-ai` transport (not litellm's anthropic `/v1/messages`). Routing Gemini would need a small wire-id bridge (anthropic transport + a gateway model alias); **deferred** to a possible future harness milestone, out of M7 scope. | -| D6 | Target names | ksvc `serverless-harness`, namespace `default` (F3) — parameterized as `KSVC`/`NS` in `lib.sh`. | -| D7 | Credentials | (Re)create the `llm-credentials` secret from the operator's env at setup; **no secrets in the repo**. | -| D8 | E3/E4 assertions | **Conversational recall**, not tool-dependent: the model must echo an earlier-turn fact (a planted token). No sandbox tool needed, so model/tool availability is irrelevant to these gates. | +| # | Decision | Choice | +| --- | ----------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | +| D1 | Driver style | **Bash, extending `smoke.sh`** (kubectl + curl + jq), under `deploy/knative/`. Cluster ops are native to kubectl/curl; `smoke.sh` already does ~80% of the choreography; no new toolchain. | +| D2 | E1 cost metric | **Sampled pod-seconds**: poll `kubectl get pods -l serving.knative.dev/service=serverless-harness -n default` every `SAMPLE_INTERVAL` (default 5s); pod-seconds = Σ(running harness pods × interval). Run the same workload once at `min-scale=1`, once at `min-scale=0`. | +| D3 | E1 pass criterion | **serverless ≤ 0.6 × persistent** pod-seconds (≥40% reduction) over the idle-heavy pattern, AND report both absolute numbers + ratio. Magnitude depends on the idle/retention ratio (documented); the gate is directional + a clear margin. | +| D4 | Scope | **One milestone**, one branch, shared setup + `lib.sh` + one `EXPERIMENTS.md`. | +| D5 | Experiment model | Set **`SH_MODEL=claude-haiku-4-5`** (dash-form, anthropic) on the ksvc — cheap/fast for the many real turns. The M6 `requireModel` guard protects against a bad id. **Note:** non-anthropic models (e.g. Gemini) through the litellm gateway are **not** a config-only switch — the gateway bridge keeps the model's pi-ai `api`, and `gemini-2.0-flash` is registered under `google` with the native `google-generative-ai` transport (not litellm's anthropic `/v1/messages`). Routing Gemini would need a small wire-id bridge (anthropic transport + a gateway model alias); **deferred** to a possible future harness milestone, out of M7 scope. | +| D6 | Target names | ksvc `serverless-harness`, namespace `default` (F3) — parameterized as `KSVC`/`NS` in `lib.sh`. | +| D7 | Credentials | (Re)create the `llm-credentials` secret from the operator's env at setup; **no secrets in the repo**. | +| D8 | E3/E4 assertions | **Conversational recall**, not tool-dependent: the model must echo an earlier-turn fact (a planted token). No sandbox tool needed, so model/tool availability is irrelevant to these gates. | --- @@ -96,6 +98,7 @@ deploy/knative/ ``` ### 4.1 Setup/harden (`run-experiments.sh` preamble + `lib.sh`) + - Resolve `NS=default`, `KSVC=serverless-harness`; derive the Kourier base URL + Host header exactly as `smoke.sh` does today. - `require_secret`: if `llm-credentials` is absent, create it from `ANTHROPIC_API_KEY` / @@ -104,6 +107,7 @@ deploy/knative/ - Smoke-check one real `/turn` before running experiments (fail fast if the deploy is unhealthy). ### 4.2 E1 — economics (`e1-economics.sh`) + - Workload: `E1_TURNS` (default 5) short `/turn` calls to one session, separated by `E1_IDLE` (default 120s) idle gaps (≫ the 30s retention, so serverless scales down between). - For `MIN in 1 0`: `set_min_scale($MIN)`; start a background sampler (`harness_pod_count` every @@ -112,6 +116,7 @@ deploy/knative/ `serverless ≤ 0.6 × persistent`. ### 4.3 E3 — mobility (`e3-mobility.sh`) + - Create a session; turn 1 plants a fact (`"Remember the code word: ZEBRA42."`); a few more turns. - Force a fresh instance: `set_min_scale(0)` + `wait_scale_to_zero` (assert the pod is gone), so the next request cold-starts a **new** pod. @@ -119,12 +124,14 @@ deploy/knative/ (instance B reconstructed context from the Redis log — not from in-pod memory). ### 4.4 E4 — recovery (`e4-recovery.sh`) + - Create a session; complete turns t1, t2, t3 (each returns 200; the log is flushed per `turn_end`). - `force_kill_pod` (`kubectl delete pod -l … --force --grace-period=0`) mid-session. - Next turn: `"List everything we've discussed so far."`. **PASS** when the response/log reflects t1–t3 (zero completed-turn loss) and the turn succeeds on the freshly-started pod. ### 4.5 Results (`EXPERIMENTS.md`) + `run-experiments.sh` appends a dated section: the E1 table (persistent/serverless pod-seconds + ratio + verdict) and E3/E4 `PASS`/`FAIL` with the asserted token / turn evidence. Committed. @@ -145,26 +152,26 @@ ratio + verdict) and E3/E4 `PASS`/`FAIL` with the asserted token / turn evidence ## 6. Deviations from the parent plan (Tasks 17, 19, 20) -| Plan | This spec | Why | -|---|---|---| -| Python drivers (`requests`, `subprocess`) | Bash extending `smoke.sh` | Matches the existing deploy scripts; no Python toolchain in a TS repo. | -| ksvc `pi-harness` in `kagenti-system` | ksvc `serverless-harness` in `default` | The real deploy (F3). | -| E1 "count pod startTimes" | Sampled pod-seconds (Σ running × interval) | startTime count measures pod *count*, not idle *cost*; sampling integrates running time. | -| E3 `__dump_context__` equality of reconstructed context | Conversational recall of a planted token on a fresh pod | Fidelity (context equality) is already proven by M5 parity + M6/E2; M7's E3 is the *mobility* half — does a new instance actually answer from the log. | -| E5 reprise on cluster | — | E5 fully covered in-process by M6. | +| Plan | This spec | Why | +| ------------------------------------------------------- | ------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------ | +| Python drivers (`requests`, `subprocess`) | Bash extending `smoke.sh` | Matches the existing deploy scripts; no Python toolchain in a TS repo. | +| ksvc `pi-harness` in `kagenti-system` | ksvc `serverless-harness` in `default` | The real deploy (F3). | +| E1 "count pod startTimes" | Sampled pod-seconds (Σ running × interval) | startTime count measures pod _count_, not idle _cost_; sampling integrates running time. | +| E3 `__dump_context__` equality of reconstructed context | Conversational recall of a planted token on a fresh pod | Fidelity (context equality) is already proven by M5 parity + M6/E2; M7's E3 is the _mobility_ half — does a new instance actually answer from the log. | +| E5 reprise on cluster | — | E5 fully covered in-process by M6. | --- ## 7. Residual risks -| Risk | Impact | Mitigation | -|------|--------|------------| -| Cluster/gateway flakiness (cold-start latency, gateway rate limits/timeouts). | Spurious FAIL. | Generous curl timeouts (ksvc `timeoutSeconds:300`); retry the setup smoke-check; drivers are re-runnable. | -| E1 magnitude depends on the idle/retention ratio; with small idle gaps the difference shrinks. | Weak/ambiguous result. | Defaults `E1_IDLE=120s` ≫ 30s retention; params tunable; report absolute numbers so the result is interpretable even if the 0.6 gate is borderline. | -| Sampled pod-seconds is approximate (5s granularity; may miss sub-interval scale events). | Small measurement error. | Acceptable for a cost *proxy*; interval tunable; both configs sampled identically so bias cancels in the ratio. | -| `claude-haiku-4-5` may not reliably echo the planted token for E3/E4. | Recall assertion flaky. | Prompts explicitly instruct verbatim recall; assertion greps a distinctive token (`ZEBRA42`); these are conversational (no tool needed, D8). Fall back to a stricter prompt if needed. | -| `scale-to-zero-pod-retention-period:30s` / `stable-window` tuning affects timing. | Waits mis-timed. | `wait_scale_to_zero` polls with a timeout (as `smoke.sh` does); thresholds derived from the ksvc annotations, not hard-coded guesses. | -| Refactoring `smoke.sh` into `lib.sh` could regress it. | Breaks the M4 smoke. | `smoke.sh` is the regression anchor — it must still pass post-refactor (§5). | +| Risk | Impact | Mitigation | +| ---------------------------------------------------------------------------------------------- | ------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| Cluster/gateway flakiness (cold-start latency, gateway rate limits/timeouts). | Spurious FAIL. | Generous curl timeouts (ksvc `timeoutSeconds:300`); retry the setup smoke-check; drivers are re-runnable. | +| E1 magnitude depends on the idle/retention ratio; with small idle gaps the difference shrinks. | Weak/ambiguous result. | Defaults `E1_IDLE=120s` ≫ 30s retention; params tunable; report absolute numbers so the result is interpretable even if the 0.6 gate is borderline. | +| Sampled pod-seconds is approximate (5s granularity; may miss sub-interval scale events). | Small measurement error. | Acceptable for a cost _proxy_; interval tunable; both configs sampled identically so bias cancels in the ratio. | +| `claude-haiku-4-5` may not reliably echo the planted token for E3/E4. | Recall assertion flaky. | Prompts explicitly instruct verbatim recall; assertion greps a distinctive token (`ZEBRA42`); these are conversational (no tool needed, D8). Fall back to a stricter prompt if needed. | +| `scale-to-zero-pod-retention-period:30s` / `stable-window` tuning affects timing. | Waits mis-timed. | `wait_scale_to_zero` polls with a timeout (as `smoke.sh` does); thresholds derived from the ksvc annotations, not hard-coded guesses. | +| Refactoring `smoke.sh` into `lib.sh` could regress it. | Breaks the M4 smoke. | `smoke.sh` is the regression anchor — it must still pass post-refactor (§5). | --- @@ -178,4 +185,4 @@ the cluster-level economics, mobility, and recovery evidence. --- -*Assisted-By: Claude (Anthropic AI) * +_Assisted-By: Claude (Anthropic AI) _ diff --git a/docs/specs/2026-06-26-harness-lockdown-design.md b/docs/specs/2026-06-26-harness-lockdown-design.md index e055cf1..7e82460 100644 --- a/docs/specs/2026-06-26-harness-lockdown-design.md +++ b/docs/specs/2026-06-26-harness-lockdown-design.md @@ -4,20 +4,20 @@ Version: 1.0 — June 26, 2026 Status: Design (approved for implementation planning) Scope: How the **harness** (the "brain" — Pi runtime + our wrapper, the component that builds prompts, calls the LLM provider, and writes the durable Redis log) is contained, given that it -holds the only secret worth defending (the provider key) and that tool execution is *not* +holds the only secret worth defending (the provider key) and that tool execution is _not_ guaranteed to be redirected to the sandbox. The thesis: the harness needs **no L7 egress proxy** — it has no model-controlled egress surface — so it is defended by making any local execution in it **unrewarding and unable to phone home**, not by mediating egress it never makes. -Milestone relationship: **Refines the harness portion of parent M7/M8.** It *lightens* M7 (no +Milestone relationship: **Refines the harness portion of parent M7/M8.** It _lightens_ M7 (no egress proxy for the harness) and sets up M8 (the provider-key injector, specified separately next). Parent design: [Zero-Trust, Multi-Agent Extensions to the Serverless Harness](../../../docs/research/2026-06-18-zero-trust-multiagent-harness-extension.md) — §2 spine, §2.2 load-bearing claims, §3.1 inference broker, §4.1 invariants. Builds on / consumes: M2 ([`K8sSandboxClient`](2026-06-17-m2-k8s-sandbox-client-design.md)), M3 ([persistent channel](2026-06-17-m3-persistent-channel-design.md)), M4 ([Knative wrapper](2026-06-17-m4-knative-serverless-wrapper-design.md)). Sibling: [M13 — Generalized Credentialed Egress](2026-06-19-m13-generalized-credentialed-egress-design.md) (the **sandbox**'s heavyweight egress plane). This design explicitly argues that apparatus does **not** extend to the harness. > **Implementation status (issue #66, 2026-07-07).** The **egress invariant** this design enforces: -> *the harness may reach only the **LLM inference** endpoint and the **sandbox** control channel; all +> _the harness may reach only the **LLM inference** endpoint and the **sandbox** control channel; all > other outbound traffic — git, arbitrary web, MCP/tool calls — flows through the sandbox, the single -> controlled I/O surface.* Two Z2 controls now back it in the tree: +> controlled I/O surface._ Two Z2 controls now back it in the tree: > **(1) Fail-closed redirection (Layer 1)** — implemented and verified on `main`: `kubectlExecInPod` > and the persistent channel reject on any child error / abort / timeout with **no local branch**, and > the tool ops propagate exec rejections directly. There is no local-exec fallback on a routing error. @@ -53,8 +53,8 @@ Verification of `K8sSandboxClient` (the `@sh/k8s-sandbox` extension) shows tool **Pi's native `LocalShell` / `LocalFileSystem` tools stand** — the harness then executes model-directed commands locally. - **And it overrides a known list of 7 tools.** Pi contains local-execution paths that bypass the - Operations seam — the build already had to special-case `grep` because *"Pi's grep always spawns a - LOCAL rg"* (see `grep-tool.ts`). A future Pi version adding a tool, a subagent-spawn path, or + Operations seam — the build already had to special-case `grep` because _"Pi's grep always spawns a + LOCAL rg"_ (see `grep-tool.ts`). A future Pi version adding a tool, a subagent-spawn path, or another "spawns local X" shortcut would not be overridden and would run in the harness. So at the capability level the harness today is **"allow-list of redirected tools with a local @@ -65,8 +65,8 @@ this design closes. - A **threat model for the harness** (§2): its assets, its trust position, and exactly what model output can and cannot make it do. -- A **lock-down design** (§4) in five layers that make local execution in the harness *unrewarding* - (nothing to steal) and *unable to exfiltrate* (no route out), plus *fail-closed* redirection so +- A **lock-down design** (§4) in five layers that make local execution in the harness _unrewarding_ + (nothing to steal) and _unable to exfiltrate_ (no route out), plus _fail-closed_ redirection so accidental local execution stops happening at all. - The explicit **lightening result** (§5): the harness needs no forward proxy, no baked CA, no placeholder-swap. @@ -75,7 +75,7 @@ this design closes. ### Out of scope (later / separate) - **The provider-key injector internals** — fixed-upstream header injection, per-session inference - budget enforcement, and how the key reaches *it* (static Secret vs. SPIRE-bound). Specified in the + budget enforcement, and how the key reaches _it_ (static Secret vs. SPIRE-bound). Specified in the **next** sibling design (the M8 injector). This doc fixes only the injector's **placement** (a separate pod, §4 layer 3 rationale) and the harness-side contract (a non-secret base-URL). - **The sandbox threat model and its egress plane** — M13. Deliberately deferred per the @@ -91,33 +91,33 @@ this design closes. ### 2.1 What the harness is Trusted code operating on **untrusted data**. The data is (a) model output and (b) the durable -session log — which a prior compromised session could have poisoned. The harness is *not* a place -model-authored code is *supposed* to run; all tool execution is meant to redirect to the sandbox. +session log — which a prior compromised session could have poisoned. The harness is _not_ a place +model-authored code is _supposed_ to run; all tool execution is meant to redirect to the sandbox. ### 2.2 Assets -| Asset | Why it matters | Worst case | -|---|---|---| -| **Provider key** | The crown jewel. The harness builds prompts **and** writes the durable, append-only, broadly-replayed log. | Serialized into the log once → compromised **forever**, readable by every future harness instance, voter, and audit tool; or echoed into the conversation. | -| **Redis write access** | The log is the source of truth / the spine. | Poisoning future resumes of this and other sessions. | -| **`pods/exec` capability** | The harness drives the sandbox via `kubectl exec`. | Exec into other pods if the SA is over-broad. | -| **Network position** | Can reach provider, Redis, sandbox. | An exfil route if egress is open to the internet. | +| Asset | Why it matters | Worst case | +| -------------------------- | ---------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------- | +| **Provider key** | The crown jewel. The harness builds prompts **and** writes the durable, append-only, broadly-replayed log. | Serialized into the log once → compromised **forever**, readable by every future harness instance, voter, and audit tool; or echoed into the conversation. | +| **Redis write access** | The log is the source of truth / the spine. | Poisoning future resumes of this and other sessions. | +| **`pods/exec` capability** | The harness drives the sandbox via `kubectl exec`. | Exec into other pods if the SA is over-broad. | +| **Network position** | Can reach provider, Redis, sandbox. | An exfil route if egress is open to the internet. | ### 2.3 Adversary & influence path The adversary is **model output** (adversarial or prompt-injected) and **poisoned log content**. -The harness is trusted code, so the adversary cannot directly run code in it — *unless* a harness +The harness is trusted code, so the adversary cannot directly run code in it — _unless_ a harness code path executes model-directed work locally. The two such paths today are exactly the §1 finding: (a) the **fail-open gate**, and (b) **un-overridden Pi local-exec paths**. ### 2.4 The egress observation that lightens everything The harness makes only **fixed-destination** egress: LLM provider (inference), Redis (log), sandbox -(`kubectl exec` / channel). **None is model-controlled.** The model changes prompt *content*, never +(`kubectl exec` / channel). **None is model-controlled.** The model changes prompt _content_, never the harness's destination. The "model code curls an arbitrary host" surface — the entire justification for M13's forward proxy + baked CA + allowlist + placeholder-swap — **does not exist in the harness.** Therefore the harness must not be defended by mediating egress; it must be -defended by ensuring that *if* local code ever runs in it, that code finds **nothing to steal** and +defended by ensuring that _if_ local code ever runs in it, that code finds **nothing to steal** and has **nowhere to send it**. ### 2.5 Trust boundaries @@ -147,24 +147,24 @@ guarantees. ## 3. Key decisions -| # | Decision | Choice | -|---|----------|--------| -| H1 | Egress proxy for the harness? | **No.** The harness has no model-controlled egress; an L7 forward proxy / baked CA / placeholder-swap (M13) is unjustified here. | -| H2 | Primary defense | **Defang local execution, don't mediate egress.** Make local code unrewarding (no secret) and unable to phone home (no route), and stop it happening (fail-closed). | -| H3 | Redirect gate posture | **Fail-closed.** In the zero-trust deployment a sandbox is *required*; unresolved sandbox config hard-fails (refuse to register local tools, refuse to run a turn). Replaces the current fail-open `if (!config) return;`. | -| H4 | Provider key location | **Not in the harness container.** Key lives only in a separate **injector pod** (M8). Harness reaches the provider via a non-secret base URL pointed at the injector. | -| H5 | Exfil boundary | **Pod-level NetworkPolicy, default-deny egress.** Allow only `{API server, Redis, injector pod}`. Kernel/CNI-enforced; no public `:443`. | -| H6 | Injector placement | **Separate pod, not a same-pod sidecar.** NetworkPolicy selects pods, not containers; a same-pod sidecar with provider egress would re-grant the harness container that egress (shared netns). A separate pod makes "harness has no internet route" actually enforceable. | -| H7 | Image | **Distroless-node (or scratch+node).** No shell/coreutils → Pi's `LocalShell` and "spawn local rg" paths fail at `ENOENT`. | -| H8 | Pod hardening | `runAsNonRoot`, `allowPrivilegeEscalation:false`, `capabilities.drop:[ALL]`, `readOnlyRootFilesystem:true` (+ emptyDir `/tmp`), `seccompProfile:RuntimeDefault`. | -| H9 | RBAC | **Least-privilege SA.** Keep `pods/exec` (the harness needs it to drive the sandbox) but scope it to **sandbox pods only** (dedicated namespace + label selector); no `get secrets`, no exec elsewhere. | -| H10 | Honesty about residue | Distroless does **not** disable Node's own `fs`/`net`/`child_process`; the NetworkPolicy (H5), not the image, is the exfil control. Layers are complementary; none is sufficient alone. | +| # | Decision | Choice | +| --- | ----------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| H1 | Egress proxy for the harness? | **No.** The harness has no model-controlled egress; an L7 forward proxy / baked CA / placeholder-swap (M13) is unjustified here. | +| H2 | Primary defense | **Defang local execution, don't mediate egress.** Make local code unrewarding (no secret) and unable to phone home (no route), and stop it happening (fail-closed). | +| H3 | Redirect gate posture | **Fail-closed.** In the zero-trust deployment a sandbox is _required_; unresolved sandbox config hard-fails (refuse to register local tools, refuse to run a turn). Replaces the current fail-open `if (!config) return;`. | +| H4 | Provider key location | **Not in the harness container.** Key lives only in a separate **injector pod** (M8). Harness reaches the provider via a non-secret base URL pointed at the injector. | +| H5 | Exfil boundary | **Pod-level NetworkPolicy, default-deny egress.** Allow only `{API server, Redis, injector pod}`. Kernel/CNI-enforced; no public `:443`. | +| H6 | Injector placement | **Separate pod, not a same-pod sidecar.** NetworkPolicy selects pods, not containers; a same-pod sidecar with provider egress would re-grant the harness container that egress (shared netns). A separate pod makes "harness has no internet route" actually enforceable. | +| H7 | Image | **Distroless-node (or scratch+node).** No shell/coreutils → Pi's `LocalShell` and "spawn local rg" paths fail at `ENOENT`. | +| H8 | Pod hardening | `runAsNonRoot`, `allowPrivilegeEscalation:false`, `capabilities.drop:[ALL]`, `readOnlyRootFilesystem:true` (+ emptyDir `/tmp`), `seccompProfile:RuntimeDefault`. | +| H9 | RBAC | **Least-privilege SA.** Keep `pods/exec` (the harness needs it to drive the sandbox) but scope it to **sandbox pods only** (dedicated namespace + label selector); no `get secrets`, no exec elsewhere. | +| H10 | Honesty about residue | Distroless does **not** disable Node's own `fs`/`net`/`child_process`; the NetworkPolicy (H5), not the image, is the exfil control. Layers are complementary; none is sufficient alone. | --- ## 4. The lock-down (five layers, by leverage) -### Layer 1 — Fail-closed redirection *(core; highest leverage, near-free)* +### Layer 1 — Fail-closed redirection _(core; highest leverage, near-free)_ In the zero-trust deployment, treat the sandbox as **required**: @@ -177,9 +177,9 @@ In the zero-trust deployment, treat the sandbox as **required**: accident," closing the §1 fail-open hole at its root. This is a deployment-mode posture, not a Pi change: local tools remain the legitimate default for -non-sandboxed local runs; the *zero-trust harness deployment* refuses them. +non-sandboxed local runs; the _zero-trust harness deployment_ refuses them. -### Layer 2 — Secret-free harness container *(core)* +### Layer 2 — Secret-free harness container _(core)_ - The provider key is **never mounted** into the harness container (no env, no file). It lives only in the injector pod (M8). @@ -187,15 +187,15 @@ non-sandboxed local runs; the *zero-trust harness deployment* refuses them. the OpenAI-compatible base URL) pointed at the injector. That value is not a secret. - **Mechanism:** Kubernetes scopes env and volume mounts per container; a Secret not mounted into the harness container is genuinely absent from its `/proc/self/environ`. Secret-free is achieved by - *not mounting*, nothing more exotic. + _not mounting_, nothing more exotic. - **Invariant (parent §4.1):** a red-team `grep` over harness env + reconstructed prompt + the Redis log finds no provider key. Local code that does run finds nothing worth taking. -### Layer 3 — Default-deny egress NetworkPolicy *(core; the exfil boundary)* +### Layer 3 — Default-deny egress NetworkPolicy _(core; the exfil boundary)_ - Default-deny egress on the harness pod; **allow only**: K8s API server (for `pods/exec`), Redis, - and the injector pod. **No `0.0.0.0/0:443`.** *(This is the **M8 end-state**; the shipped v1 below - is a documented intermediate that still has an external hop because the injector is not yet built.)* + and the injector pod. **No `0.0.0.0/0:443`.** _(This is the **M8 end-state**; the shipped v1 below + is a documented intermediate that still has an external hop because the injector is not yet built.)_ - This is the control that actually closes exfil: even Node `fs`/`net` code running locally in the harness (which distroless cannot stop, H10) has **nowhere to send data**. - **H6 dependency:** because NetworkPolicy is per-pod and a pod shares one netns, the injector must @@ -219,7 +219,7 @@ non-sandboxed local runs; the *zero-trust harness deployment* refuses them. tunnel cannot exist. Even at v1, default-deny still blocks all lateral in-cluster movement, all non-allowlisted ports, and any plaintext exfil. -### Layer 4 — Distroless image + hardened securityContext *(defense-in-depth)* +### Layer 4 — Distroless image + hardened securityContext _(defense-in-depth)_ - **Distroless-node**: no shell, no coreutils, no `rg`. Pi's un-overridden `LocalShell` / local-`rg` paths fail closed at `ENOENT` instead of executing. @@ -229,14 +229,14 @@ non-sandboxed local runs; the *zero-trust harness deployment* refuses them. but `fs.readFile` (on read-only mounts) and `net`/`http` still work. Distroless kills the shell/coreutils class; the NetworkPolicy kills the network class. Necessary, not sufficient, alone. -### Layer 5 — Least-privilege RBAC *(defense-in-depth)* +### Layer 5 — Least-privilege RBAC _(defense-in-depth)_ - The harness SA **needs** `create pods/exec` to drive the sandbox; don't drop the token. - Scope it to exec **only on sandbox pods** (dedicated namespace + label selector). No `get/list - secrets`, no exec on arbitrary pods. This constrains the one genuine privilege the harness holds — +secrets`, no exec on arbitrary pods. This constrains the one genuine privilege the harness holds — the thing local-exec'd code would most want to abuse. -*Optional belt-and-suspenders:* if Pi's API allows **removing** (not just overriding) the local +_Optional belt-and-suspenders:_ if Pi's API allows **removing** (not just overriding) the local Operations implementations at startup, do it — then the local handlers aren't present in-process, partially addressing H10. Only if clean; don't fight the framework. @@ -250,7 +250,7 @@ partially addressing H10. Only if clean; don't fight the framework. - **No baked CA / TLS interception** of the harness's traffic. - **No placeholder-swap, no allowlist machinery, no per-host policy** at the harness. -Those exist to tame *arbitrary model-controlled egress*, which the harness does not have (§2.4). +Those exist to tame _arbitrary model-controlled egress_, which the harness does not have (§2.4). **Buys:** @@ -261,7 +261,7 @@ Those exist to tame *arbitrary model-controlled egress*, which the harness does - A clean, kernel-enforced network boundary that does not depend on enumerating Pi's every local-exec path (a losing game across version bumps, as `grep` already showed). -**Honest residue (§8 expands):** L2's guarantee bounds *raw-key exfiltration*, not *use* — a popped +**Honest residue (§8 expands):** L2's guarantee bounds _raw-key exfiltration_, not _use_ — a popped harness can still ask the injector to proxy provider calls while alive. And the injector pod becomes a higher-value target (holds the key, sees prompts); that is the subject of the next spec. @@ -269,12 +269,12 @@ a higher-value target (holds the key, sees prompts); that is the subject of the ## 6. Failure modes & caveats -- **Sandbox config missing/unreachable** → harness **refuses to run** (L1), loud error; it does *not* +- **Sandbox config missing/unreachable** → harness **refuses to run** (L1), loud error; it does _not_ silently fall back to local tools. (This is the intended new behavior.) - **A future Pi tool/path not overridden** → it may attempt local execution, but: no shell (L4), no secret (L2), no internet route (L3), and scoped RBAC (L5). Defanged rather than perfectly prevented. -- **Harness process compromise (RCE in trusted code)** → can *use* the injector (proxy provider +- **Harness process compromise (RCE in trusted code)** → can _use_ the injector (proxy provider calls) and write Redis while alive, but cannot exfiltrate the raw key or reach arbitrary hosts. Blast radius bounded to the pod's lifetime and its allowed in-cluster peers. - **NetworkPolicy not enforced by the CNI** → L3 silently no-ops. Deployment must assert a @@ -295,20 +295,20 @@ The harness lock-down passes when, on a Kind cluster with the zero-trust harness injector). 3. **No exfil route:** from inside the harness container, an outbound connection to an arbitrary public host on `:443` **fails**; connections to `{API server, Redis, injector}` succeed. - *This gate requires a policy-enforcing CNI and therefore runs on **OCP (OVN-Kubernetes)**, not the + _This gate requires a policy-enforcing CNI and therefore runs on **OCP (OVN-Kubernetes)**, not the Kind base — kindnet does not enforce egress policy, so there the manifest is present but a no-op (Z2 §6). Note that **pre-M8**, arbitrary-host `:443` still succeeds by construction (the LLM is external); it is the injector end-state that makes it fail. The CNI-independent half — that the manifest has the right **shape** (default-deny egress, correct Knative pod selector, only the `{DNS, Redis, HTTPS}` allowlist, and both kustomizations wiring it in) — is asserted in CI by - `packages/knative-server/test/harness-egress-policy.test.ts`.* + `packages/knative-server/test/harness-egress-policy.test.ts`._ - **Liveness under the policy:** the OCP gate must also confirm the harness pod reaches and **holds `Ready`** with the policy applied — including a **scale-from-zero** — so that the Knative **queue-proxy** sidecar's own control-plane egress needs are covered by the allowlist. Queue-proxy shares the pod netns, so a default-deny egress that omits one of its egress peers can silently **starve the sidecar** and surface only as a failed scale-up (the pod never goes `Ready`), not as an obvious connection error. Assert readiness + a successful cold-start request, not just - LLM/Redis/API reachability. (Autoscaler metrics are *scraped* — ingress to the pod — and this + LLM/Redis/API reachability. (Autoscaler metrics are _scraped_ — ingress to the pod — and this policy is `Egress`-only, so that path is unaffected; the check guards against any egress peer we have not enumerated.) 4. **Local-exec defanged:** a forced un-routed local-exec attempt (e.g. a tool that would spawn a @@ -327,7 +327,7 @@ The harness lock-down passes when, on a Kind cluster with the zero-trust harness visibility in one pod is the cost of the clean boundary. Mitigations (trusted code, mTLS/SPIFFE gating, budget enforcement, key rotation) are the **next spec**'s job. 3. **`use` vs. `exfiltrate` residue.** L2 bounds raw-key leakage, not key use by a live compromised - harness. Accepted; it is a large reduction *because the log is durable*, and a full fix would + harness. Accepted; it is a large reduction _because the log is durable_, and a full fix would require attesting the harness process itself (out of scope). 4. **CNI dependency.** L3 is only as real as the cluster's NetworkPolicy enforcement. 5. **Distroless operational cost.** No shell in the image complicates in-container debugging; use an @@ -348,4 +348,4 @@ The harness lock-down passes when, on a Kind cluster with the zero-trust harness --- -*Assisted-By: Claude (Anthropic AI) * +_Assisted-By: Claude (Anthropic AI) _ diff --git a/docs/specs/2026-06-26-identity-spine-design.md b/docs/specs/2026-06-26-identity-spine-design.md index 62e2ea2..db9b585 100644 --- a/docs/specs/2026-06-26-identity-spine-design.md +++ b/docs/specs/2026-06-26-identity-spine-design.md @@ -15,9 +15,9 @@ Parent design: [Zero-Trust, Multi-Agent Extensions](../../../docs/research/2026- Consumed by: [Z2 Harness Lock-Down](2026-06-26-harness-lockdown-design.md) (harness SPIFFE id for mTLS to the injector; `pods/exec`-only RBAC), [Z3 Inference Injector](2026-06-26-inference-injector-design.md) (mTLS peer authz), [Z4 MCP code-mode](2026-06-18-m10-mcp-code-mode-design.md) + [Z5 Generalized Egress](2026-06-19-m13-generalized-credentialed-egress-design.md) (per-user `(actor SVID ⊕ subject)` resolution). > **The one-sentence thesis.** Per-user isolation in a shared namespace, under scale-to-zero, -> reduces to: *a trusted orchestrator mints a per-session SPIFFE identity with the user in the +> reduces to: _a trusted orchestrator mints a per-session SPIFFE identity with the user in the > attested path, derived from a durable binding it alone can write, and reconstructs it on every -> wake* — and everything else (harness, sandbox) is built untrusted around that. +> wake_ — and everything else (harness, sandbox) is built untrusted around that. --- @@ -27,7 +27,7 @@ Consumed by: [Z2 Harness Lock-Down](2026-06-26-harness-lockdown-design.md) (harn Give every session a **per-session, mesh-verifiable identity bound to its human user**, such that the egress planes (Z3 inference, Z5 sandbox) can resolve credentials per-user **without** any -workload being able to assert or spoof *who it is* — including when **many users share one +workload being able to assert or spoof _who it is_ — including when **many users share one namespace** and when sessions **scale to zero** between turns. ### In scope @@ -36,7 +36,7 @@ namespace** and when sessions **scale to zero** between turns. - The **per-session identity model**: SPIFFE id with the user in the attested path; why per-namespace identity is insufficient — §3. - The portable **`CredentialInjector` interface** + the **kagenti binding** (SPIRE + Istio ambient - + waypoint + AuthBridge) — §4. + - waypoint + AuthBridge) — §4. - The **orchestrator**: responsibilities, lifecycle, and the integrity invariant that only it can mint identity — §5. - The **authoritative binding store**, integrity-protected and **separate from the model-influenced @@ -47,7 +47,7 @@ namespace** and when sessions **scale to zero** between turns. ### Out of scope (Z5 / separate) - **The per-user external-credential store** (linking/consent/rotation/revocation of GitHub PATs, - OAuth grants, etc.). Z1 issues *identity*; Z5 resolves *credentials* keyed by that identity. Z1 + OAuth grants, etc.). Z1 issues _identity_; Z5 resolves _credentials_ keyed by that identity. Z1 records the dependency's shape (§12) but does not build it. - **The egress proxies themselves** — the inference injector (Z3) and the sandbox waypoint/forward proxy (Z5). Z1 defines the interface they implement. @@ -62,15 +62,15 @@ namespace** and when sessions **scale to zero** between turns. Per-user isolation requires a control tier **above** both the brain and the hands. The identity plane introduces it explicitly: -| Tier | Component | Trust | `pods/create`? | Mints identity? | Holds secrets? | -|---|---|---|---|---|---| -| **Control** | **Orchestrator** | trusted; **not** model-influenced | **yes (sole)** | **yes (sole)** | no | -| **Brain** | Harness | semi-trusted (untrusted *data*) | **no** | no | no (Z2) | -| **Hands** | Sandbox | untrusted (model code) | no | no | no (Z5; uses, never reads) | +| Tier | Component | Trust | `pods/create`? | Mints identity? | Holds secrets? | +| ----------- | ---------------- | --------------------------------- | -------------- | --------------- | -------------------------- | +| **Control** | **Orchestrator** | trusted; **not** model-influenced | **yes (sole)** | **yes (sole)** | no | +| **Brain** | Harness | semi-trusted (untrusted _data_) | **no** | no | no (Z2) | +| **Hands** | Sandbox | untrusted (model code) | no | no | no (Z5; uses, never reads) | The orchestrator is the new element. It is shared, long-lived control-plane infrastructure (in kagenti: the operator extended with a **Session controller**), not a per-session pod and **not** -scale-to-zero. It provisions *identity*; it never touches per-user secrets, which keeps it out of +scale-to-zero. It provisions _identity_; it never touches per-user secrets, which keeps it out of the credential blast radius even though it is the identity-side crown jewel (§9). --- @@ -132,20 +132,20 @@ interface CredentialInjector: The interface does not depend on SPIRE, Envoy, or Keycloak — a non-kagenti cluster can implement it with a minimal secret-holding sidecar behind the same contract. This is what keeps serverless-harness -portable (parent §2.3) and what makes Z3 (provider key) and Z5 (per-user egress) two *implementations* +portable (parent §2.3) and what makes Z3 (provider key) and Z5 (per-user egress) two _implementations_ of one idea rather than two designs. ### 4.2 kagenti reference binding -| Concern | Binding | -|---|---| -| Identity issuance | **SPIRE** issues per-session SVIDs (§3.3); `spiffe-helper` delivers them in-pod. | -| Transport identity | **Istio ambient ztunnel** — L4 mTLS, verified `source.principal` per pod. | +| Concern | Binding | +| ------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | +| Identity issuance | **SPIRE** issues per-session SVIDs (§3.3); `spiffe-helper` delivers them in-pod. | +| Transport identity | **Istio ambient ztunnel** — L4 mTLS, verified `source.principal` per pod. | | L7 injection point | **waypoint** (Envoy) — the only place headers/credentials are set. ztunnel (L4) cannot inject. **One shared waypoint per namespace** serves many per-session identities; it keys on `source.principal`, so it does **not** churn per user. | -| Resolution | **AuthBridge** ext-proc reads the verified identity → `(actor SVID ⊕ subject)` → mint (RFC 8693) or fetch stored grant (Z5) → inject. | +| Resolution | **AuthBridge** ext-proc reads the verified identity → `(actor SVID ⊕ subject)` → mint (RFC 8693) or fetch stored grant (Z5) → inject. | -The harness path (Z3) is the *trivial* implementation: identity gates mTLS to the injector; the -credential is a non-per-user provider key. The sandbox path (Z5) is the *full* implementation: +The harness path (Z3) is the _trivial_ implementation: identity gates mTLS to the injector; the +credential is a non-per-user provider key. The sandbox path (Z5) is the _full_ implementation: identity → user → that user's grant. --- @@ -157,9 +157,9 @@ identity → user → that user's grant. - **Create:** authenticate the user (or accept the gateway/Keycloak assertion) → allocate `session-id` → write the **authoritative `session-id → user` binding** (§6) → mint the per-session SPIRE registration with the user in the attested path → provision the identity-bearing sandbox pod. - *This is the one point a live user identity is required.* + _This is the one point a live user identity is required._ - **Wake:** receive the wake signal for `session-id` → read the binding from the durable store → - reconstruct the SPIRE entry + sandbox pod with the correct identity → hand the harness a *reference* + reconstruct the SPIRE entry + sandbox pod with the correct identity → hand the harness a _reference_ to connect to. **No live user re-auth** — wake is unattended (§7). - **Idle (scale-to-zero):** tear down the sandbox pod and reap the SPIRE entry; **keep** the binding. - **End / GC:** delete binding + entry; TTL-reap abandoned sessions. @@ -193,11 +193,11 @@ If these share one harness-writable structure, a compromised harness could rewri - The binding lives in an **orchestrator-owned, integrity-protected store** (Redis ACL-scoped key / a Session CR / Keycloak) that the **harness cannot write**. -- The log may still *reference* identity for **audit** (§4.1 holds for audit), but it is **not +- The log may still _reference_ identity for **audit** (§4.1 holds for audit), but it is **not authoritative** for resolution. This sharpens parent §4.1 ("identity referenced by SPIFFE string only"): a SPIFFE string in the log is -fine *as audit*, but the **resolution-authoritative** binding is orchestrator-owned and write-isolated +fine _as audit_, but the **resolution-authoritative** binding is orchestrator-owned and write-isolated from the log. --- @@ -222,7 +222,7 @@ Two properties make this clean: - **No credential is ever cached in-pod** — resolution fetches the grant fresh at egress every time, so teardown loses nothing credential-related. - **Stored-grant resolution (Z5 §4.2) means the user may be offline** at wake. Scale-to-zero is the - *payoff case* for choosing stored grants over live user tokens: a session can act as Alice while + _payoff case_ for choosing stored grants over live user tokens: a session can act as Alice while Alice is offline. A live-token design would break here. A long-idle wake may find the stored grant expired/revoked → egress returns a clean auth error (Z5: @@ -232,31 +232,31 @@ A long-idle wake may find the stored grant expired/revoked → egress returns a ## 8. Key decisions -| # | Decision | Choice | -|---|----------|--------| -| ID1 | Identity granularity | **Per-session**, finer than namespace/SA. The namespace is not the isolation boundary (§3.2). | -| ID2 | User binding | **User in the attested SVID path**, set by the orchestrator at mint time; the resolver reads it — no asserted header (§3.1). | -| ID3 | Issuance | **SPIRE per-pod registration** (kagenti-native; `spiffe-helper` already present); per-session SA as fallback (§3.3). | -| ID4 | Interface | **Abstract `CredentialInjector`** (`identity → egress injection`); kagenti binding = SPIRE + ambient + waypoint + AuthBridge (§4). Z3 and Z5 are implementations. | -| ID5 | Minting authority | **Orchestrator only** creates session pods and SPIRE entries; harness has no `pods/create` (§5.2). | -| ID6 | Binding store | **Orchestrator-owned, integrity-protected, separate from the harness-writable log** (§6). | -| ID7 | Scale-to-zero | **Reconstruct identity on wake from the durable binding; reap on idle.** No per-session mesh state while sleeping (§7). | -| ID8 | Live user auth | **At session creation only.** Wake is unattended, trusting the durable binding + stored grant (§7). | -| ID9 | Harness reframing | Harness gets a SPIFFE id but **no egress waypoint** (parent M7 reframed; Z2 §2.4). The per-user egress plane is the sandbox's (Z5). | +| # | Decision | Choice | +| --- | -------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| ID1 | Identity granularity | **Per-session**, finer than namespace/SA. The namespace is not the isolation boundary (§3.2). | +| ID2 | User binding | **User in the attested SVID path**, set by the orchestrator at mint time; the resolver reads it — no asserted header (§3.1). | +| ID3 | Issuance | **SPIRE per-pod registration** (kagenti-native; `spiffe-helper` already present); per-session SA as fallback (§3.3). | +| ID4 | Interface | **Abstract `CredentialInjector`** (`identity → egress injection`); kagenti binding = SPIRE + ambient + waypoint + AuthBridge (§4). Z3 and Z5 are implementations. | +| ID5 | Minting authority | **Orchestrator only** creates session pods and SPIRE entries; harness has no `pods/create` (§5.2). | +| ID6 | Binding store | **Orchestrator-owned, integrity-protected, separate from the harness-writable log** (§6). | +| ID7 | Scale-to-zero | **Reconstruct identity on wake from the durable binding; reap on idle.** No per-session mesh state while sleeping (§7). | +| ID8 | Live user auth | **At session creation only.** Wake is unattended, trusting the durable binding + stored grant (§7). | +| ID9 | Harness reframing | Harness gets a SPIFFE id but **no egress waypoint** (parent M7 reframed; Z2 §2.4). The per-user egress plane is the sandbox's (Z5). | --- ## 9. Threat model & blast radius (honest) - **The orchestrator is the identity-side crown jewel.** If compromised, it can mint an SVID bound to - *any* user and drive egress to **use** that user's stored grant — i.e. impersonate any user. It + _any_ user and drive egress to **use** that user's stored grant — i.e. impersonate any user. It never holds the raw secrets (those stay in the cred store, injected at the waypoint), but it can cause their use. This mirrors the injector's role on the provider-key side (Z3): the trust core is two small, non-model-influenced concentration points. - **Mitigations:** minimal non-model-influenced surface; strong RBAC; an **audit trail of every identity-minting action** (who minted `…/user/alice/session/X`, from which authenticated request); - and ideally **separation of duties** — the component that *authenticates the user* distinct from the - one that *mints entries*, so neither alone can impersonate. + and ideally **separation of duties** — the component that _authenticates the user_ distinct from the + one that _mints entries_, so neither alone can impersonate. - **The fatal anti-pattern (explicitly forbidden):** shared namespace SA + a `subject` header for per-user routing → any session spoofs any user → cross-user credential theft (§3.2). - **Spoofing the SVID:** a workload cannot obtain another session's SVID without compromising SPIRE or @@ -325,4 +325,4 @@ resolver, in **one shared namespace**: --- -*Assisted-By: Claude (Anthropic AI) * +_Assisted-By: Claude (Anthropic AI) _ diff --git a/docs/specs/2026-06-26-inference-injector-design.md b/docs/specs/2026-06-26-inference-injector-design.md index edb63e6..5dc16d8 100644 --- a/docs/specs/2026-06-26-inference-injector-design.md +++ b/docs/specs/2026-06-26-inference-injector-design.md @@ -13,13 +13,13 @@ harness lock-down (this date's sibling) depends on. Refines parent §3.1 from a **separate shared gateway pod** (the NetworkPolicy-granularity reason, harness-lockdown H6). Parent design: [Zero-Trust, Multi-Agent Extensions](../../../docs/research/2026-06-18-zero-trust-multiagent-harness-extension.md) — §2.2 the harness-holds-no-key claim, §3.1 inference broker, §4.1 log invariants, §2.3 portable-core/kagenti-binding split. Depends on / pairs with: [Harness Lock-Down Design](2026-06-26-harness-lockdown-design.md) — H4 (key not in harness; non-secret base URL), H5 (only the injector has public egress), H6 (separate pod), §8 (the injector is the high-value target this spec must own). -Sibling: [M13 — Generalized Credentialed Egress](2026-06-19-m13-generalized-credentialed-egress-design.md) — the **sandbox**'s egress plane. This injector is deliberately *not* that: no baked CA, no placeholder-swap, no allowlist-per-host policy. +Sibling: [M13 — Generalized Credentialed Egress](2026-06-19-m13-generalized-credentialed-egress-design.md) — the **sandbox**'s egress plane. This injector is deliberately _not_ that: no baked CA, no placeholder-swap, no allowlist-per-host policy. > **Why this is light.** The harness explicitly points its provider base URL at the injector, so > there is **no deception and no TLS interception** (contrast M13's sandbox forward proxy + baked > CA). The injector is a path-preserving host+auth rewrite for a small static set of providers — > not a policy engine. The zero-trust property it delivers ("the harness holds no key") comes from -> *where the key lives*, not from elaborate request mediation. +> _where the key lives_, not from elaborate request mediation. --- @@ -65,7 +65,7 @@ non-model-influenced component — so the harness lock-down's default-deny egres ### 2.1 What the injector is **Trusted code that is NOT influenced by model output.** It does not build prompts, parse model -output, or run model-authored code. It transits an inference request whose *body* originated in the +output, or run model-authored code. It transits an inference request whose _body_ originated in the harness (and ultimately reflects model/context content), but it treats that body as **opaque bytes** — it neither inspects nor logs it. Its only inputs it acts on are the routing header and the peer identity. @@ -75,24 +75,26 @@ peer identity. The harness lock-down (§8) names the injector the high-value target by design — concentration is the cost of the clean boundary: -| Asset | Exposure | -|---|---| -| **Provider keys** (all providers) | At rest in the injector pod's Secret mounts; the single rotation/audit point. | -| **Public internet egress** | The only component allowed to reach the provider hosts on `:443`. | -| **Prompt bodies in transit** | It transits every inference request — same visibility the provider already has. Retains **none** of it (§7, §8). | +| Asset | Exposure | +| --------------------------------- | ---------------------------------------------------------------------------------------------------------------- | +| **Provider keys** (all providers) | At rest in the injector pod's Secret mounts; the single rotation/audit point. | +| **Public internet egress** | The only component allowed to reach the provider hosts on `:443`. | +| **Prompt bodies in transit** | It transits every inference request — same visibility the provider already has. Retains **none** of it (§7, §8). | ### 2.3 What it defends — and what it does not **Defends (structurally):** + - The key never enters the harness, so it can never reach the durable log (parent §4.1). Achieved by - *where the key lives*, mTLS-gated access, and the injector always overwriting client auth. + _where the key lives_, mTLS-gated access, and the injector always overwriting client auth. - Exfil to arbitrary hosts from the harness path: the injector forwards only to the static provider upstreams (its own egress allowlist); the harness pod has no other public route (lock-down H5). **Does NOT defend (honest residue):** + - **A live-compromised harness can use the injector** — it holds a valid mTLS identity, so it can - ask the injector to proxy provider calls while it is alive. The injector hides the *raw key*, not - *use* of it. `x-sh-session` is attribution, **not** an authorization control. + ask the injector to proxy provider calls while it is alive. The injector hides the _raw key_, not + _use_ of it. `x-sh-session` is attribution, **not** an authorization control. - **A compromised injector is game over** for keys — it is the concentration point. Mitigations: trusted code, minimal surface, no body retention, mTLS-gated ingress, least-privilege, rotation. @@ -115,19 +117,19 @@ cost of the clean boundary: ## 3. Key decisions -| # | Decision | Choice | -|---|----------|--------| -| I1 | Form | **Minimal purpose-built reverse proxy** (e.g. Go `httputil.ReverseProxy`). Portable core, no mesh dependency; owns streaming + audit. (Envoy and AuthBridge-reuse considered and rejected as heavier for this job.) | -| I2 | Topology | **One shared, long-lived gateway Deployment** (separate pod, parent option (b)); stateless in v1 → multi-replica trivially; not scale-to-zero (stable egress point). | -| I3 | Provider breadth | **Multi-provider via a static provider table.** Each entry: upstream host, auth scheme (location + name + format), key Secret ref. | -| I4 | Provider selection | **Per-request header `x-sh-provider`.** The harness knows its provider (`SH_MODEL_PROVIDER`) and sets the header; unknown value → `400`. | -| I5 | Credential placement | **Static K8s Secret(s) mounted only to the injector pod.** Never in the harness. SPIRE-bound fetch is the documented upgrade (§13). | -| I6 | Credential handling | **Strip then set.** The injector removes any client-supplied auth (`Authorization`, `x-api-key`, `x-goog-api-key`, version headers it owns) and sets the real credential, so the harness cannot influence or smuggle it. | -| I7 | Harness↔injector transport | **mTLS in v1.** Mutually authenticated; injector authorizes known harness SPIFFE identities. kagenti binding: Istio ambient + SPIRE. NetworkPolicy is defense-in-depth, not the sole control. | -| I8 | Provider TLS | **Injector originates fresh TLS** to the real provider. **No baked CA, no interception** — the harness explicitly targets the injector, so there is no deception (contrast M13). | -| I9 | Body handling | **Opaque, streaming, path-preserving.** Bodies and request paths pass through unmodified; the injector mutates only headers + upstream host. No body parse, no body log. | -| I10 | Budget | **None in v1.** Turn-boundary voter (M5/M6) keeps it. Hard per-session cap is the upgrade (§13). | -| I11 | Audit | **Metadata only:** session, provider, model id (from the `x-sh-model` header, so the body stays opaque), request/response sizes, status, timestamp. Never key, never body. | +| # | Decision | Choice | +| --- | -------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | +| I1 | Form | **Minimal purpose-built reverse proxy** (e.g. Go `httputil.ReverseProxy`). Portable core, no mesh dependency; owns streaming + audit. (Envoy and AuthBridge-reuse considered and rejected as heavier for this job.) | +| I2 | Topology | **One shared, long-lived gateway Deployment** (separate pod, parent option (b)); stateless in v1 → multi-replica trivially; not scale-to-zero (stable egress point). | +| I3 | Provider breadth | **Multi-provider via a static provider table.** Each entry: upstream host, auth scheme (location + name + format), key Secret ref. | +| I4 | Provider selection | **Per-request header `x-sh-provider`.** The harness knows its provider (`SH_MODEL_PROVIDER`) and sets the header; unknown value → `400`. | +| I5 | Credential placement | **Static K8s Secret(s) mounted only to the injector pod.** Never in the harness. SPIRE-bound fetch is the documented upgrade (§13). | +| I6 | Credential handling | **Strip then set.** The injector removes any client-supplied auth (`Authorization`, `x-api-key`, `x-goog-api-key`, version headers it owns) and sets the real credential, so the harness cannot influence or smuggle it. | +| I7 | Harness↔injector transport | **mTLS in v1.** Mutually authenticated; injector authorizes known harness SPIFFE identities. kagenti binding: Istio ambient + SPIRE. NetworkPolicy is defense-in-depth, not the sole control. | +| I8 | Provider TLS | **Injector originates fresh TLS** to the real provider. **No baked CA, no interception** — the harness explicitly targets the injector, so there is no deception (contrast M13). | +| I9 | Body handling | **Opaque, streaming, path-preserving.** Bodies and request paths pass through unmodified; the injector mutates only headers + upstream host. No body parse, no body log. | +| I10 | Budget | **None in v1.** Turn-boundary voter (M5/M6) keeps it. Hard per-session cap is the upgrade (§13). | +| I11 | Audit | **Metadata only:** session, provider, model id (from the `x-sh-model` header, so the body stays opaque), request/response sizes, status, timestamp. Never key, never body. | --- @@ -164,7 +166,7 @@ gemini: { host: generativelanguage.googleapis.com, auth: {loc: header, name: The table, not code branches, encodes the difference. - **`extra`** carries non-secret required headers (e.g. `anthropic-version`). - Keys are **mounted, not embedded**: each `secret` ref is a K8s Secret mounted to the injector pod - only; the table holds the *reference*, never the value. + only; the table holds the _reference_, never the value. --- @@ -175,10 +177,10 @@ gemini: { host: generativelanguage.googleapis.com, auth: {loc: header, name: principals (SPIRE-issued). **Portable core:** the contract is "mTLS + identity-based authz"; a non-kagenti cluster supplies its own mutual-TLS mechanism. - **NetworkPolicy (defense-in-depth):** only harness pods may reach the injector; the injector's - egress is allowed only to the provider upstream hosts. mTLS authorizes *who*; NetworkPolicy bounds - *reachability* — both, not either. + egress is allowed only to the provider upstream hosts. mTLS authorizes _who_; NetworkPolicy bounds + _reachability_ — both, not either. - **`x-sh-session` is attribution, not authorization.** It feeds audit (and the future budget cap). - A valid-mTLS harness is trusted to *use* inference; the session header does not gate that (§2.3 + A valid-mTLS harness is trusted to _use_ inference; the session header does not gate that (§2.3 residue). --- @@ -308,4 +310,4 @@ The injector passes when, on a Kind cluster with the locked-down harness deploym --- -*Assisted-By: Claude (Anthropic AI) * +_Assisted-By: Claude (Anthropic AI) _ diff --git a/docs/specs/2026-06-26-leaf-session-backend-capability-charter.md b/docs/specs/2026-06-26-leaf-session-backend-capability-charter.md index 7b22184..a9f9b7c 100644 --- a/docs/specs/2026-06-26-leaf-session-backend-capability-charter.md +++ b/docs/specs/2026-06-26-leaf-session-backend-capability-charter.md @@ -2,8 +2,8 @@ Version: 1.0 — June 26, 2026 Status: Charter (roadmap anchor; informs Phase-2 priority and the MVP target) -Scope: Answers one question — *does the serverless-harness design generalize beyond a single -pipeline?* — by testing it against three independent agentic-pipeline archetypes, and uses the +Scope: Answers one question — _does the serverless-harness design generalize beyond a single +pipeline?_ — by testing it against three independent agentic-pipeline archetypes, and uses the answer to **repoint** the Phase-2 roadmap and define the MVP target. This is a positioning/charter doc, not a milestone implementation spec. Source of truth for numbering: [Milestone Registry](README.md). This charter **reprioritizes** the @@ -12,9 +12,9 @@ Builds on the built base: M1 (Redis session), M2/M3 (sandbox), M4 (Knative scale (checkpoint/resume), M6 (runtime model selection). Relates to: [Z1 Identity Spine](2026-06-26-identity-spine-design.md), [Z2 Harness Lock-Down](2026-06-26-harness-lockdown-design.md), [Z3 Inference Injector](2026-06-26-inference-injector-design.md), [Z5 Generalized Egress](2026-06-19-m13-generalized-credentialed-egress-design.md) — this charter says **which of these the MVP needs and which it defers.** -> **One-line finding.** Three independent agentic pipelines all share the same shape — *a +> **One-line finding.** Three independent agentic pipelines all share the same shape — _a > deterministic, non-LLM orchestrator dispatching parameterized agent **leaf** sessions, with -> structured artifacts, checkpoint/resume, and human/audit gates* — and **none uses recursive +> structured artifacts, checkpoint/resume, and human/audit gates_ — and **none uses recursive > subagents.** So the harness's highest-value role is to be an excellent **leaf-session backend > invoked by an arbitrary external orchestrator**, not to build an orchestrator or a subagent system. @@ -24,23 +24,23 @@ Relates to: [Z1 Identity Spine](2026-06-26-identity-spine-design.md), [Z2 Harnes The credential-plane re-examination produced a coherent Z-track, but the priority order was unanchored. To anchor it, we tested the harness design against **three independent, real agentic -pipelines** (kept name-free here as archetypes). The test: *can the harness run all three, and what -does each actually need?* +pipelines** (kept name-free here as archetypes). The test: _can the harness run all three, and what +does each actually need?_ ## 2. Evidence — three archetypes -| | **Archetype A**: parallel-fan-out analysis | **Archetype B**: sequential role loop | **Archetype C**: scheduled ingestion | -|---|---|---|---| -| Shape | candidate → review → validate, batched | hypothesis → design → execute → analyze, iterated | fetch → dedup → filter → refine, repeated | -| Orchestrator | deterministic code (prepare/run/finalize + audit gates) | deterministic state machine (atomic checkpoint/resume) | **scheduler (cron) + events** + deterministic stages | -| Agent topology | flat pool of **parallel worker** leaf-sessions | **two sequential roles**, ~2 LLM calls/iteration | deterministic stages + **LLM leaf calls** | -| **Recursive subagents?** | **No** | **No** (parallelism = isolated workspaces) | **No** | -| Model tiering | per-phase (cheap triage / strong validate) | per-role (strong plan / cheaper execute) | light (filter/refine) | -| State / artifacts | structured JSON, shared volume, resumable | schema-governed JSON, atomic checkpoint, cross-iteration memory | JSON + cursors, **git-backed** | -| Human gates | minimal (blocker only) | **explicit** approve/reject/abort + auto-mode | **issue/comment** gates | -| Start signal | invocation | invocation | **cron + event** | -| Egress | model API, VCS, package registries | model API + optional secondary endpoint | model API, content/source APIs, VCS push | -| Sandbox/tools | indexers, search, build toolchains | workspace isolation, build/run, patch apply | API clients, containerized filter | +| | **Archetype A**: parallel-fan-out analysis | **Archetype B**: sequential role loop | **Archetype C**: scheduled ingestion | +| ------------------------ | ------------------------------------------------------- | --------------------------------------------------------------- | ---------------------------------------------------- | +| Shape | candidate → review → validate, batched | hypothesis → design → execute → analyze, iterated | fetch → dedup → filter → refine, repeated | +| Orchestrator | deterministic code (prepare/run/finalize + audit gates) | deterministic state machine (atomic checkpoint/resume) | **scheduler (cron) + events** + deterministic stages | +| Agent topology | flat pool of **parallel worker** leaf-sessions | **two sequential roles**, ~2 LLM calls/iteration | deterministic stages + **LLM leaf calls** | +| **Recursive subagents?** | **No** | **No** (parallelism = isolated workspaces) | **No** | +| Model tiering | per-phase (cheap triage / strong validate) | per-role (strong plan / cheaper execute) | light (filter/refine) | +| State / artifacts | structured JSON, shared volume, resumable | schema-governed JSON, atomic checkpoint, cross-iteration memory | JSON + cursors, **git-backed** | +| Human gates | minimal (blocker only) | **explicit** approve/reject/abort + auto-mode | **issue/comment** gates | +| Start signal | invocation | invocation | **cron + event** | +| Egress | model API, VCS, package registries | model API + optional secondary endpoint | model API, content/source APIs, VCS push | +| Sandbox/tools | indexers, search, build toolchains | workspace isolation, build/run, patch apply | API clients, containerized filter | ### 2.1 The invariants across all three @@ -49,9 +49,9 @@ does each actually need?* 2. **Agents are parameterized leaf invocations** — `(model, inputs) → structured output`. 3. **No recursive subagents in currently-running code.** "Parallel agentic work" is a deterministic orchestrator fanning out leaf sessions, or **isolated workspaces** — never agent-spawns-agent. - *(Verified nuance: A **deliberately rejected** agent-managed subagents for bulk fan-out; B has + _(Verified nuance: A **deliberately rejected** agent-managed subagents for bulk fan-out; B has **planned-but-not-yet-merged** clean-context subagents for per-scope exploration and per-arm - isolation. See [Archetypes & Requirements](2026-06-26-pipeline-archetypes-requirements.md) §6.)* + isolation. See [Archetypes & Requirements](2026-06-26-pipeline-archetypes-requirements.md) §6.)_ 4. **Per-role / per-phase model tiering**, multi-provider, env-selected. 5. **Structured, schema-governed artifacts** on a durable store, with **audit/coverage/ledger** and **checkpoint/resume + incremental memory**. @@ -64,17 +64,17 @@ does each actually need?* > The serverless-harness is a **scale-to-zero, durable, sandboxed, model-tiered leaf-session > backend**, invoked over a stable contract (HTTP/CLI) by an **arbitrary external deterministic -> orchestrator** — a state machine, a staged script, *or a CI/cron scheduler* — offering optional +> orchestrator** — a state machine, a staged script, _or a CI/cron scheduler_ — offering optional > primitives for **checkpoint/resume, human-gates, workspace isolation, and credentialed egress**. ### 3.1 Non-goals (what the harness must NOT impose) - **It does not host or supply the orchestrator.** All three bring their own; the harness is - *invoked*, it does not drive. (This resolves the earlier "where does the orchestrator run" fork: + _invoked_, it does not drive. (This resolves the earlier "where does the orchestrator run" fork: **workers-only**.) - **It does not impose a domain artifact store.** A uses a shared volume, C uses a git repo. The harness provides **session/turn durability** (its Redis log); the **domain artifacts stay the - orchestrator's** (file/volume/git). *Session durability ≠ domain artifacts.* (This resolves the + orchestrator's** (file/volume/git). _Session durability ≠ domain artifacts._ (This resolves the "artifact exchange" fork: don't force Redis.) - **It does not provide recursive subagents.** None of the three need them. @@ -82,21 +82,21 @@ does each actually need?* ## 4. Capability set → status -| Capability | Needed by | Status / home | -|---|---|---| -| Sandboxed tool execution | A, B, C | ✅ built (M2/M3) | -| Scale-to-zero session runtime | A, B, C | ✅ built (M4) | -| Durable session + **checkpoint/resume** | A, B, C | ✅ built (M1/M5) — **promote to a first-class API** | -| Per-phase/role **model tiering** | A, B, C | ✅ built (M6 runtime model input) | -| **Leaf-session invocation contract** (run-to-completion, parameterized inputs, structured output) | A, B, C | ⭐ **new — the core MVP capability** | -| **Workspace isolation** (per-session worktree / results-dir / CoW) | A, B | ⭐ **new — promote** | -| **Human-gate primitive** (pause → structured summary → approve/reject/abort; auto-mode) | B, C | ⭐ **new — promote (post-MVP)** | -| **Trigger/start on-ramp** (HTTP/event/cron) | C | ⭐ **new — promote (post-MVP)**; Knative is already HTTP-triggered | -| Model-key injection (hide provider key) | A, B, C | �🔹 keep-light (Z3); MVP may keep key in env | -| Per-source **credentialed egress** (VCS, content APIs, registries) | A, C | 🔹 keep-light (Z5) when sources are authenticated | -| Clean-context subtask delegation (a leaf spawns a fresh-context child) | B (planned) | ⭐ **re-entrant contract** (design property) — core need met for free | -| Recursive-subagent **extras** (lineage, budget, policy, mail) | B (later) | ⏸️ **defer (Z6 extras)** — not the MVP | -| Per-user identity / multi-tenant isolation | **none** (all are per-project automations) | ⏸️ **defer (Z1)** — only if multi-tenant hosting becomes a goal | +| Capability | Needed by | Status / home | +| ------------------------------------------------------------------------------------------------- | ------------------------------------------ | --------------------------------------------------------------------- | +| Sandboxed tool execution | A, B, C | ✅ built (M2/M3) | +| Scale-to-zero session runtime | A, B, C | ✅ built (M4) | +| Durable session + **checkpoint/resume** | A, B, C | ✅ built (M1/M5) — **promote to a first-class API** | +| Per-phase/role **model tiering** | A, B, C | ✅ built (M6 runtime model input) | +| **Leaf-session invocation contract** (run-to-completion, parameterized inputs, structured output) | A, B, C | ⭐ **new — the core MVP capability** | +| **Workspace isolation** (per-session worktree / results-dir / CoW) | A, B | ⭐ **new — promote** | +| **Human-gate primitive** (pause → structured summary → approve/reject/abort; auto-mode) | B, C | ⭐ **new — promote (post-MVP)** | +| **Trigger/start on-ramp** (HTTP/event/cron) | C | ⭐ **new — promote (post-MVP)**; Knative is already HTTP-triggered | +| Model-key injection (hide provider key) | A, B, C | �🔹 keep-light (Z3); MVP may keep key in env | +| Per-source **credentialed egress** (VCS, content APIs, registries) | A, C | 🔹 keep-light (Z5) when sources are authenticated | +| Clean-context subtask delegation (a leaf spawns a fresh-context child) | B (planned) | ⭐ **re-entrant contract** (design property) — core need met for free | +| Recursive-subagent **extras** (lineage, budget, policy, mail) | B (later) | ⏸️ **defer (Z6 extras)** — not the MVP | +| Per-user identity / multi-tenant isolation | **none** (all are per-project automations) | ⏸️ **defer (Z1)** — only if multi-tenant hosting becomes a goal | --- @@ -124,13 +124,13 @@ is the **next step** after this charter — see Open Questions §8. ## 6. Explicit deferrals (with rationale) - **Z6 recursive subagents — extras deferred; core met by a re-entrant contract.** Zero of three - archetypes spawn subagents in running code, and A *deliberately rejected* agent-managed subagents + archetypes spawn subagents in running code, and A _deliberately rejected_ agent-managed subagents for **bulk** fan-out (a deterministic pool over leaf sessions is better). **But demand is not zero:** B has planned (not-yet-merged) **clean-context subagents** for per-scope exploration and per-arm isolation — a leaf agent offloading a focused subtask to a fresh-context child. That need is satisfied **for free by a re-entrant leaf-session contract**: a leaf session dispatches a **child** leaf session via the same contract, getting clean context + its own sandbox by - construction. So the MVP contract must be **re-entrancy-friendly**, and only Z6's *extras* — + construction. So the MVP contract must be **re-entrancy-friendly**, and only Z6's _extras_ — parent/child lineage, budget propagation, sandbox policy, inter-agent messaging — actually defer. Bulk fan-out stays deterministic; recursion is opt-in via the same contract. - **Z1 per-user identity — deferred.** All three are **per-project automations**, not multi-user @@ -145,16 +145,16 @@ plane ahead of demand. ## 7. Key decisions -| # | Decision | Choice | -|---|----------|--------| -| G1 | Harness role | **Leaf-session backend**, invoked by an external orchestrator; not an orchestrator itself. | -| G2 | Orchestrator placement | **External / workers-only.** The harness never hosts the orchestrator. | -| G3 | Artifact store | **Not imposed.** Harness owns session/turn durability; domain artifacts stay the orchestrator's (volume/git). | -| G4 | Subagents | **Core met by a re-entrant contract; Z6 extras deferred.** Bulk fan-out is deterministic over leaf sessions; clean-context delegation (B, planned) = a leaf dispatching a child leaf via the same contract. Only lineage/budget/policy/mail defer. | -| G5 | Identity | **Per-user identity deferred (Z1).** Single-tenant MVP; key in env. | -| G6 | Core MVP capability | The **leaf-session invocation contract** (run-to-completion, parameterized, structured output) + **workspace isolation**. | -| G7 | Model tiering | Reuse **runtime model selection (M6)**; injector (Z3) is a later hardening, not MVP. | -| G8 | Promote post-MVP | **Human-gate** and **trigger/cron** primitives (needed by B and C, not by the first slice). | +| # | Decision | Choice | +| --- | ---------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| G1 | Harness role | **Leaf-session backend**, invoked by an external orchestrator; not an orchestrator itself. | +| G2 | Orchestrator placement | **External / workers-only.** The harness never hosts the orchestrator. | +| G3 | Artifact store | **Not imposed.** Harness owns session/turn durability; domain artifacts stay the orchestrator's (volume/git). | +| G4 | Subagents | **Core met by a re-entrant contract; Z6 extras deferred.** Bulk fan-out is deterministic over leaf sessions; clean-context delegation (B, planned) = a leaf dispatching a child leaf via the same contract. Only lineage/budget/policy/mail defer. | +| G5 | Identity | **Per-user identity deferred (Z1).** Single-tenant MVP; key in env. | +| G6 | Core MVP capability | The **leaf-session invocation contract** (run-to-completion, parameterized, structured output) + **workspace isolation**. | +| G7 | Model tiering | Reuse **runtime model selection (M6)**; injector (Z3) is a later hardening, not MVP. | +| G8 | Promote post-MVP | **Human-gate** and **trigger/cron** primitives (needed by B and C, not by the first slice). | --- @@ -164,13 +164,13 @@ plane ahead of demand. (parallel leaf sessions + tiering + workspace + structured artifacts); C (scheduled ingestion) is simpler but pulls in the trigger primitive early. Pick one and define the thin vertical slice. 2. **Artifact-store span.** The contract says "don't impose," but the harness still needs a - *convention* for handing a leaf session its inputs and collecting its structured output. Define + _convention_ for handing a leaf session its inputs and collecting its structured output. Define the minimal envelope (e.g. an inputs dir + a results path on a mounted volume) without owning the domain store. 3. **Human-gate vs scale-to-zero.** A gate that waits for human approval implies a session that sleeps indefinitely — a natural fit for scale-to-zero + durable resume, but the gate's pending-state must live in durable state, not a live pod. Design when promoting the gate. -4. **Trigger placement.** Does the cron/event trigger belong *in* the harness, or stay the +4. **Trigger placement.** Does the cron/event trigger belong _in_ the harness, or stay the orchestrator's (e.g. GitHub Actions) with the harness only exposing the HTTP contract? Leaning: keep triggers external; the harness exposes the contract. @@ -185,4 +185,4 @@ plane ahead of demand. --- -*Assisted-By: Claude (Anthropic AI) * +_Assisted-By: Claude (Anthropic AI) _ diff --git a/docs/specs/2026-06-26-mvp-leaf-session-contract-design.md b/docs/specs/2026-06-26-mvp-leaf-session-contract-design.md index 4dc7345..90261fe 100644 --- a/docs/specs/2026-06-26-mvp-leaf-session-contract-design.md +++ b/docs/specs/2026-06-26-mvp-leaf-session-contract-design.md @@ -7,9 +7,10 @@ for an external deterministic orchestrator**, using the parallel-fan-out archety the **leaf-session invocation contract** — parameterized, run-to-completion, structured-output, workspace-isolated, on scale-to-zero infra — and nothing domain-specific. Realizes: [Capability Charter](2026-06-26-leaf-session-backend-capability-charter.md) §5 (MVP core) -+ §7 (G1, G6). Single-tenant, key-in-env. -Builds on (reuse, no new design): M2/M3 (sandbox), M4 (Knative scale-to-zero), M5 (checkpoint/resume), M6 (runtime model selection). -Defers (per charter): Z1 identity, Z3 injector, Z5 egress, Z6 *extras* (the core clean-context-subagent need is met by a **re-entrant contract**, §2.5); human-gates; triggers; real candidate-generation and PoC/exploit stages. + +- §7 (G1, G6). Single-tenant, key-in-env. + Builds on (reuse, no new design): M2/M3 (sandbox), M4 (Knative scale-to-zero), M5 (checkpoint/resume), M6 (runtime model selection). + Defers (per charter): Z1 identity, Z3 injector, Z5 egress, Z6 _extras_ (the core clean-context-subagent need is met by a **re-entrant contract**, §2.5); human-gates; triggers; real candidate-generation and PoC/exploit stages. > **Superseded in part by P1 (July 2, 2026).** The **volume envelope** (file-based `inputsRef` / > `resultRef` on the `/work` PVC) is replaced by an inline-inputs + inline/Redis-verdict contract in @@ -18,8 +19,8 @@ Defers (per charter): Z1 identity, Z3 injector, Z5 egress, Z6 *extras* (the core > structured output, workspace isolation, re-entrant resume) are unchanged — only the transport. > **What this slice is NOT.** It is not the real analysis pipeline. The agentic task is a -> representative **stub** (flag a pattern in a file) so the slice measures *the contract and the -> integration seam*, not domain logic. The real candidate-generation and validation stages plug into +> representative **stub** (flag a pattern in a file) so the slice measures _the contract and the +> integration seam_, not domain logic. The real candidate-generation and validation stages plug into > the same contract later. --- @@ -83,8 +84,8 @@ request **blocks until the session reaches a terminal state**; the orchestrator ### 2.2 Run-to-completion ("job mode") Distinct from the interactive `runTurn`: the harness seeds the agent with a fixed prompt — -*"process the item in `inputs_ref` against the repo at `workspace_ref`; emit your verdict by calling -`submit_verdict`"* — and runs the agent loop **autonomously to completion** (until `submit_verdict` +_"process the item in `inputs_ref` against the repo at `workspace_ref`; emit your verdict by calling +`submit_verdict`"_ — and runs the agent loop **autonomously to completion** (until `submit_verdict` is called, or `max_turns`/timeout). Reuses `runTurn` internals; adds the completion loop. ### 2.3 Structured output (the verdict) @@ -113,14 +114,14 @@ overwrites — so **retry = re-invoke**. The harness writes `result_ref` fresh e ### 2.5 Re-entrancy (design property, not MVP scope) The contract must be designed so a **leaf session can itself invoke `/runs`** to dispatch a -**child** leaf session. This is *not built or tested in the MVP* (the MVP is single-level: +**child** leaf session. This is _not built or tested in the MVP_ (the MVP is single-level: orchestrator → leaf), but the contract must **not preclude** it, because it is how the one genuine near-term subagent need is met: a leaf agent that wants a fresh-context subtask (see [Archetypes & Requirements](2026-06-26-pipeline-archetypes-requirements.md) §6, archetype B's planned `explore`/per-arm subagents) simply dispatches a child leaf — which gets a clean context and its own sandbox **by construction**, with no separate subagent runtime. Concretely, the MVP keeps the contract self-contained (no caller-identity assumption that only an external orchestrator may call -it) so re-entrancy is a later increment, not a redesign. The Z6 *extras* — parent/child lineage +it) so re-entrancy is a later increment, not a redesign. The Z6 _extras_ — parent/child lineage (`parent_session_id`), budget propagation, sandbox policy, inter-agent messaging — are deferred. --- @@ -132,7 +133,7 @@ external orchestrator driver (deterministic; brings its own logic) │ for each of N items: POST /runs {session_id, model, inputs_ref, result_ref, workspace_ref} ▼ (N concurrent requests) Knative harness service ── scales out to N pods, scales to zero when idle (M4) - │ job-mode: seed prompt → run agent to completion (M6 model) + │ job-mode: seed prompt → run agent to completion (M6 model) ▼ sandbox (M2/M3): read workspace_ref (RO), use tools (read/grep), reason with model │ agent calls submit_verdict(args) @@ -147,17 +148,17 @@ orchestrator: read all result_refs → retry any failed/missing (re-invoke) → ## 4. Components -| # | Component | New / reuse | -|---|---|---| -| 1 | **Invocation contract + `/runs` endpoint** | ⭐ new (the core) | -| 2 | **Job-mode completion loop** (autonomous run to verdict) | ⭐ new (wraps `runTurn`) | -| 3 | **`submit_verdict` tool + schema validation + result_ref write** | ⭐ new | -| 4 | **Minimal orchestrator driver** (fan-out / collect / retry / coverage audit) | ⭐ new (test stand-in) | -| 5 | **Fixture repo + verdict schema** | ⭐ new (small) | -| 6 | Sandbox tool execution | ♻️ M2/M3 | -| 7 | Knative parallel + scale-to-zero | ♻️ M4 | -| 8 | Per-call model selection | ♻️ M6 | -| 9 | Checkpoint/resume (verify a restarted session resumes) | ♻️ M5 | +| # | Component | New / reuse | +| --- | ---------------------------------------------------------------------------- | ------------------------ | +| 1 | **Invocation contract + `/runs` endpoint** | ⭐ new (the core) | +| 2 | **Job-mode completion loop** (autonomous run to verdict) | ⭐ new (wraps `runTurn`) | +| 3 | **`submit_verdict` tool + schema validation + result_ref write** | ⭐ new | +| 4 | **Minimal orchestrator driver** (fan-out / collect / retry / coverage audit) | ⭐ new (test stand-in) | +| 5 | **Fixture repo + verdict schema** | ⭐ new (small) | +| 6 | Sandbox tool execution | ♻️ M2/M3 | +| 7 | Knative parallel + scale-to-zero | ♻️ M4 | +| 8 | Per-call model selection | ♻️ M6 | +| 9 | Checkpoint/resume (verify a restarted session resumes) | ♻️ M5 | --- @@ -244,4 +245,4 @@ PVC + the fixture repo: --- -*Assisted-By: Claude (Anthropic AI) * +_Assisted-By: Claude (Anthropic AI) _ diff --git a/docs/specs/2026-06-26-pipeline-archetypes-requirements.md b/docs/specs/2026-06-26-pipeline-archetypes-requirements.md index db6ac2a..5994fb4 100644 --- a/docs/specs/2026-06-26-pipeline-archetypes-requirements.md +++ b/docs/specs/2026-06-26-pipeline-archetypes-requirements.md @@ -15,19 +15,19 @@ Anchors: [Capability Charter](2026-06-26-leaf-session-backend-capability-charter ## 1. The three archetypes at a glance -| | **A — Parallel fan-out analysis** | **B — Iterative role-based loop** | **C — Scheduled ingestion & filtering** | -|---|---|---|---| -| Purpose | scan a corpus → produce candidate findings → review → validate each | form a hypothesis → design+run a controlled experiment → extract principles → iterate | fetch items from external sources on a schedule → dedup → filter for relevance → refine | -| Orchestrator | deterministic staged scripts (`prepare→run→finalize`) with audit gates | deterministic state machine with atomic checkpoint/resume | a scheduler (cron) + event triggers + deterministic stages | -| Agent topology | flat pool of **parallel worker** leaf-sessions (one per batch/finding) | **two sequential roles** (planner, executor); ~2 LLM calls per iteration | deterministic stages + **single-call LLM leaf** filters | -| Parallelism | high (tens–hundreds of workers), bounded by a concurrency cap + per-class cap | low; isolation via per-condition **workspaces** (not agents) | low; per-source | -| Model tiering | per-phase: cheap pre-filter, stronger validation | per-role: stronger planner, cheaper executor | light (a small/fast model for filtering) | -| Human gates | minimal (blocker intervention only) | **explicit** approve / reject / abort + auto-mode | **issue/comment**-driven include / skip / clarify | -| State / artifacts | structured JSON batches + verdicts on a shared volume; resumable | schema-governed JSON (state, ledger, principles); atomic checkpoint; cross-iteration memory | JSON + incremental cursors; committed to a **git data store** | -| Start signal | invocation (a target corpus) | invocation (a target + spec) | **cron schedule + events** | -| Egress | model API, source control, package registries | model API (+ optional secondary endpoint) | model API, external content/source APIs, source-control push | -| Sandbox/tooling | symbol indexing, fast search, per-language build toolchains (for validation) | workspace isolation, build/run, patch capture & apply | external API clients; a containerized filter step | -| Scale profile | bursty, LLM-cost-dominated, long idle | long-running, human-paced, intermittent | periodic (scheduled), short bursts | +| | **A — Parallel fan-out analysis** | **B — Iterative role-based loop** | **C — Scheduled ingestion & filtering** | +| ----------------- | ----------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------- | +| Purpose | scan a corpus → produce candidate findings → review → validate each | form a hypothesis → design+run a controlled experiment → extract principles → iterate | fetch items from external sources on a schedule → dedup → filter for relevance → refine | +| Orchestrator | deterministic staged scripts (`prepare→run→finalize`) with audit gates | deterministic state machine with atomic checkpoint/resume | a scheduler (cron) + event triggers + deterministic stages | +| Agent topology | flat pool of **parallel worker** leaf-sessions (one per batch/finding) | **two sequential roles** (planner, executor); ~2 LLM calls per iteration | deterministic stages + **single-call LLM leaf** filters | +| Parallelism | high (tens–hundreds of workers), bounded by a concurrency cap + per-class cap | low; isolation via per-condition **workspaces** (not agents) | low; per-source | +| Model tiering | per-phase: cheap pre-filter, stronger validation | per-role: stronger planner, cheaper executor | light (a small/fast model for filtering) | +| Human gates | minimal (blocker intervention only) | **explicit** approve / reject / abort + auto-mode | **issue/comment**-driven include / skip / clarify | +| State / artifacts | structured JSON batches + verdicts on a shared volume; resumable | schema-governed JSON (state, ledger, principles); atomic checkpoint; cross-iteration memory | JSON + incremental cursors; committed to a **git data store** | +| Start signal | invocation (a target corpus) | invocation (a target + spec) | **cron schedule + events** | +| Egress | model API, source control, package registries | model API (+ optional secondary endpoint) | model API, external content/source APIs, source-control push | +| Sandbox/tooling | symbol indexing, fast search, per-language build toolchains (for validation) | workspace isolation, build/run, patch capture & apply | external API clients; a containerized filter step | +| Scale profile | bursty, LLM-cost-dominated, long idle | long-running, human-paced, intermittent | periodic (scheduled), short bursts | --- @@ -44,7 +44,7 @@ confirmed finding (is it real, is the path reachable, can a proof be built, what dispatched by an **OS subprocess pool** with a global concurrency cap and a per-rule/class cap (to avoid API throttling), per-worker timeout, exponential-backoff retry, a circuit breaker, per-worker logs, and idempotent progress files. **Workers do not spawn child agents** — and the design comments -make this an explicit choice: the subprocess pool *replaces* "ask the main agent to spawn N +make this an explicit choice: the subprocess pool _replaces_ "ask the main agent to spawn N sub-agents in its own context." **Distinctive requirements:** batching by rule/class; per-class concurrency caps; coverage audit @@ -110,7 +110,7 @@ All three share: **checkpoint/resume or incremental cursors**. 5. **Human-in-the-loop gates** as a first-class control point (central in B and C; light in A). 6. **Code/tool sandbox** with **isolated workspaces**; the orchestrator owns its **artifact store** - (volume *or* git), not the harness. + (volume _or_ git), not the harness. 7. **Event/schedule or invocation start**; egress to a model API plus a few external services. --- @@ -121,10 +121,10 @@ All three share: splits subagent demand into two distinct patterns: - **Bulk parallel fan-out** (A's many candidates; B's arms). Here the evidence says deterministic - orchestration over **leaf sessions** is the *better* tool — A explicitly **rejected** agent-managed + orchestration over **leaf sessions** is the _better_ tool — A explicitly **rejected** agent-managed subagents for this ("…replaces ask the main agent to spawn N sub-agents in its own context"). - **Clean-context subtask delegation** (B's planned per-scope **explore** subagents, and per-arm - worktree subagents). This is a *genuine* subagent use — a leaf agent offloading a focused subtask to + worktree subagents). This is a _genuine_ subagent use — a leaf agent offloading a focused subtask to a fresh-context child. B intends it (built, not yet merged). **Conclusion for the harness:** the clean-context-subagent need is satisfied **for free by a @@ -143,90 +143,100 @@ Grouped; each requirement notes which archetype(s) need it, the harness capabili (✅ built · ⭐ new/promote · 🔹 keep-light · ⏸️ defer). "MVP" marks the thin-slice core. ### 7.1 Orchestration & control flow -| Req | Needs | Capability / status | -|---|---|---| -| Deterministic, non-LLM outer orchestrator (external to the harness) | A B C | external; harness is invoked, not orchestrator (charter G1/G2) | -| Stage sequencing with machine-checkable gates between stages | A | external orchestrator concern | -| State machine with atomic checkpoint/resume | B | external; harness offers session checkpoint ✅ (M5) | -| Scheduler (cron) + event triggers as the start signal | C | ⭐ trigger on-ramp (promote post-MVP); Knative is HTTP-triggered ✅ | -| Invocation-style start (a target + params) | A B | ✅ MVP (the invocation contract) | + +| Req | Needs | Capability / status | +| ------------------------------------------------------------------- | ----- | ------------------------------------------------------------------- | +| Deterministic, non-LLM outer orchestrator (external to the harness) | A B C | external; harness is invoked, not orchestrator (charter G1/G2) | +| Stage sequencing with machine-checkable gates between stages | A | external orchestrator concern | +| State machine with atomic checkpoint/resume | B | external; harness offers session checkpoint ✅ (M5) | +| Scheduler (cron) + event triggers as the start signal | C | ⭐ trigger on-ramp (promote post-MVP); Knative is HTTP-triggered ✅ | +| Invocation-style start (a target + params) | A B | ✅ MVP (the invocation contract) | ### 7.2 Leaf-session execution -| Req | Needs | Capability / status | -|---|---|---| -| Parameterized leaf invocation `(model, inputs) → structured output` | A B C | ⭐ MVP — the leaf-session contract | -| Run-to-completion ("job mode"), bounded by max-turns | A B | ⭐ MVP | -| Single-call LLM leaf (no agent loop) | C | ⭐ MVP (degenerate case of run-to-completion) | -| Structured output with schema validation + self-correct on mismatch | A B C | ⭐ MVP (`submit_*` tool + validation) | + +| Req | Needs | Capability / status | +| ----------------------------------------------------------------------------------- | ----------- | -------------------------------------------------------- | +| Parameterized leaf invocation `(model, inputs) → structured output` | A B C | ⭐ MVP — the leaf-session contract | +| Run-to-completion ("job mode"), bounded by max-turns | A B | ⭐ MVP | +| Single-call LLM leaf (no agent loop) | C | ⭐ MVP (degenerate case of run-to-completion) | +| Structured output with schema validation + self-correct on mismatch | A B C | ⭐ MVP (`submit_*` tool + validation) | | **Re-entrant contract** (a leaf can dispatch a child leaf = clean-context subagent) | B (planned) | ⭐ design property (non-MVP scope; do not preclude) — §6 | -| Per-leaf max-turns / tool-use caps | A B | ✅ (harness budget/turns) | +| Per-leaf max-turns / tool-use caps | A B | ✅ (harness budget/turns) | ### 7.3 Parallelism, batching, retry, coverage -| Req | Needs | Capability / status | -|---|---|---| -| Parallel worker fan-out (tens–hundreds), bounded by concurrency cap | A | ✅ Knative scale-out (M4); orchestrator sets fan-out | -| Per-class/per-rule concurrency cap (throttle avoidance) | A | external orchestrator concern | -| Batching by rule/class; configurable batch size; locality grouping | A | external orchestrator concern | -| Per-worker timeout; exponential-backoff retry; circuit breaker | A | external orchestrator concern; harness must be safe to re-invoke | -| Idempotent re-invocation (retry = re-dispatch same key) | A B | ⭐ MVP (session_id idempotency key) | -| Coverage audit (every item has a valid result; sums reconcile) | A | external orchestrator concern; harness emits per-leaf status | + +| Req | Needs | Capability / status | +| ------------------------------------------------------------------- | ----- | ---------------------------------------------------------------- | +| Parallel worker fan-out (tens–hundreds), bounded by concurrency cap | A | ✅ Knative scale-out (M4); orchestrator sets fan-out | +| Per-class/per-rule concurrency cap (throttle avoidance) | A | external orchestrator concern | +| Batching by rule/class; configurable batch size; locality grouping | A | external orchestrator concern | +| Per-worker timeout; exponential-backoff retry; circuit breaker | A | external orchestrator concern; harness must be safe to re-invoke | +| Idempotent re-invocation (retry = re-dispatch same key) | A B | ⭐ MVP (session_id idempotency key) | +| Coverage audit (every item has a valid result; sums reconcile) | A | external orchestrator concern; harness emits per-leaf status | ### 7.4 Model tiering -| Req | Needs | Capability / status | -|---|---|---| -| Per-phase / per-role model selection | A B C | ✅ runtime model input (M6) | + +| Req | Needs | Capability / status | +| ------------------------------------------------------------- | ----- | ---------------------------------------------------- | +| Per-phase / per-role model selection | A B C | ✅ runtime model input (M6) | | Multi-provider (OpenAI-compatible, Anthropic, cloud variants) | A B C | ✅ (M6); model-key injection 🔹 Z3 (env key for MVP) | -| Model resolution precedence (env > config > default) | A B | external orchestrator concern | +| Model resolution precedence (env > config > default) | A B | external orchestrator concern | ### 7.5 Artifacts, state, memory -| Req | Needs | Capability / status | -|---|---|---| -| Structured artifact I/O on a durable store **owned by the orchestrator** | A B C | harness must NOT impose a store (charter G3) | -| Shared-volume artifact handoff (inputs/results refs) | A | ⭐ MVP (volume envelope) | -| Git-backed artifact store | C | external; harness stays store-agnostic | -| Session/turn durability (resume a session) | A B | ✅ (M1/M5) — distinct from domain artifacts | -| Cross-iteration / incremental memory (principles; cursors/dedup) | B C | external orchestrator concern (domain store) | -| Schema-governed artifacts + validator | A B | ⭐ MVP for the result; broader validation external | + +| Req | Needs | Capability / status | +| ------------------------------------------------------------------------ | ----- | -------------------------------------------------- | +| Structured artifact I/O on a durable store **owned by the orchestrator** | A B C | harness must NOT impose a store (charter G3) | +| Shared-volume artifact handoff (inputs/results refs) | A | ⭐ MVP (volume envelope) | +| Git-backed artifact store | C | external; harness stays store-agnostic | +| Session/turn durability (resume a session) | A B | ✅ (M1/M5) — distinct from domain artifacts | +| Cross-iteration / incremental memory (principles; cursors/dedup) | B C | external orchestrator concern (domain store) | +| Schema-governed artifacts + validator | A B | ⭐ MVP for the result; broader validation external | ### 7.6 Human-in-the-loop -| Req | Needs | Capability / status | -|---|---|---| -| Gate primitive: pause → structured summary → approve/reject/abort | B | ⭐ promote post-MVP | -| Auto-approve mode gated by safety preconditions | B | ⭐ promote post-MVP | -| Gate-while-idle (a session may sleep awaiting approval) | B | ⭐ promote — fits scale-to-zero + durable resume | -| Event-driven gates via an external system (issues/comments) | C | external; harness exposes status | + +| Req | Needs | Capability / status | +| ----------------------------------------------------------------- | ----- | ------------------------------------------------ | +| Gate primitive: pause → structured summary → approve/reject/abort | B | ⭐ promote post-MVP | +| Auto-approve mode gated by safety preconditions | B | ⭐ promote post-MVP | +| Gate-while-idle (a session may sleep awaiting approval) | B | ⭐ promote — fits scale-to-zero + durable resume | +| Event-driven gates via an external system (issues/comments) | C | external; harness exposes status | ### 7.7 Sandbox & tooling -| Req | Needs | Capability / status | -|---|---|---| -| Sandboxed tool execution (read/search/build/run) | A B C | ✅ (M2/M3) | -| Tooled image (indexer, fast search, per-language build toolchains) | A B | ⭐ adopt real image (post-thin-slice) | -| Isolated workspace per leaf/condition (read-only mount → CoW/worktree) | A B | ⭐ MVP (read-only mount); CoW/worktree later | -| Patch capture & apply; reset between conditions | B | external orchestrator concern + sandbox | -| Process/PID limits (fork-bomb guard) | A | ✅ pod resource limits | + +| Req | Needs | Capability / status | +| ---------------------------------------------------------------------- | ----- | -------------------------------------------- | +| Sandboxed tool execution (read/search/build/run) | A B C | ✅ (M2/M3) | +| Tooled image (indexer, fast search, per-language build toolchains) | A B | ⭐ adopt real image (post-thin-slice) | +| Isolated workspace per leaf/condition (read-only mount → CoW/worktree) | A B | ⭐ MVP (read-only mount); CoW/worktree later | +| Patch capture & apply; reset between conditions | B | external orchestrator concern + sandbox | +| Process/PID limits (fork-bomb guard) | A | ✅ pod resource limits | ### 7.8 Egress & credentials -| Req | Needs | Capability / status | -|---|---|---| -| Egress to model API | A B C | 🔹 Z3 injector (env key for MVP) | -| Egress to source control / content / package APIs | A C | 🔹 Z5 (env/SSH creds for MVP) | -| Credential injection by env / mounted secret (not baked) | A B C | 🔹 Z3/Z5; env for MVP | -| No secret in logs/artifacts/prompt | A B C | ✅ invariant (Z2 §4.1 / charter) | -| Per-user credentials in a shared tenant | none | ⏸️ defer (Z1) — all three are single-tenant/per-project | + +| Req | Needs | Capability / status | +| -------------------------------------------------------- | ----- | ------------------------------------------------------- | +| Egress to model API | A B C | 🔹 Z3 injector (env key for MVP) | +| Egress to source control / content / package APIs | A C | 🔹 Z5 (env/SSH creds for MVP) | +| Credential injection by env / mounted secret (not baked) | A B C | 🔹 Z3/Z5; env for MVP | +| No secret in logs/artifacts/prompt | A B C | ✅ invariant (Z2 §4.1 / charter) | +| Per-user credentials in a shared tenant | none | ⏸️ defer (Z1) — all three are single-tenant/per-project | ### 7.9 Observability, resilience, guardrails -| Req | Needs | Capability / status | -|---|---|---| -| Per-leaf logs; structured status; failure classification | A B C | ⭐ MVP (terminal status + reasons) | -| Token/cost metrics per leaf and per run | A B | ⭐ promote (audit metadata) | -| Pre-flight credential/dependency check | B | external orchestrator concern | -| Locked-spec guardrails (immutable params, hard-fail on drift) | B | external orchestrator concern; harness enforces fail-closed config (Z2 L1) | -| Graceful partial success (continue past failed leaves) | A | external orchestrator concern | + +| Req | Needs | Capability / status | +| ------------------------------------------------------------- | ----- | -------------------------------------------------------------------------- | +| Per-leaf logs; structured status; failure classification | A B C | ⭐ MVP (terminal status + reasons) | +| Token/cost metrics per leaf and per run | A B | ⭐ promote (audit metadata) | +| Pre-flight credential/dependency check | B | external orchestrator concern | +| Locked-spec guardrails (immutable params, hard-fail on drift) | B | external orchestrator concern; harness enforces fail-closed config (Z2 L1) | +| Graceful partial success (continue past failed leaves) | A | external orchestrator concern | ### 7.10 Multi-tenancy & identity (none required today) -| Req | Needs | Capability / status | -|---|---|---| -| Per-user identity / per-session SPIFFE bound to user | none | ⏸️ defer (Z1) — pull in only for multi-tenant hosting | + +| Req | Needs | Capability / status | +| ---------------------------------------------------------- | --------------- | -------------------------------------------------------------- | +| Per-user identity / per-session SPIFFE bound to user | none | ⏸️ defer (Z1) — pull in only for multi-tenant hosting | | Recursive subagent runtime (lineage, budget, policy, mail) | B (extras only) | ⏸️ defer (Z6 extras); core covered by re-entrant contract (§6) | --- @@ -252,4 +262,4 @@ Grouped; each requirement notes which archetype(s) need it, the harness capabili --- -*Assisted-By: Claude (Anthropic AI) * +_Assisted-By: Claude (Anthropic AI) _ diff --git a/docs/specs/2026-06-27-async-leaf-completion-design.md b/docs/specs/2026-06-27-async-leaf-completion-design.md index 3123b02..95dd53f 100644 --- a/docs/specs/2026-06-27-async-leaf-completion-design.md +++ b/docs/specs/2026-06-27-async-leaf-completion-design.md @@ -41,7 +41,7 @@ completion decouples the orchestrator from task duration: - **B — iterative role-based loop**: "gate-while-idle" (a session sleeps awaiting approval) is the canonical async + scale-to-zero + durable-resume case (archetypes §7.6). Not built here, but the substrate (scale-from-queue-to-zero + resume) is exactly what B will reuse. -- **C — scheduled ingestion**: needs cron/event triggers as the *start signal*. KEDA's scaler +- **C — scheduled ingestion**: needs cron/event triggers as the _start signal_. KEDA's scaler catalog (cron, queue/stream) is precisely that on-ramp — adopting KEDA now is the production trigger substrate C will extend, without bespoke trigger code. @@ -80,16 +80,16 @@ orchestrator polls the done-marker on its own volume (or GET /runs/status?sess **Components** (each independently testable): -| Unit | Responsibility | Lives in | -|---|---|---| -| enqueue handler | validate envelope, `XADD` to `leaf-queue`, return `202` + handle | `knative-server/src/server.ts` | -| status handler | report `queued\|running\|done\|failed` from the marker (+ queue state) | `knative-server/src/server.ts` | -| `WorkQueue` | Redis Streams primitive: `ensureGroup`/`enqueue`/`claim`/`ack`/`touch`/`pending` | `packages/work-queue` (`@sh/work-queue`) | -| `leaf-job-runner` | wrapper loop: claim → `runLeaf` → classify → marker → ack/reclaim | `harness/src/leaf-job-runner.ts` | -| `classifyOutcome` | pure ack-vs-reclaim decision (see §6) | `harness/src/classify-outcome.ts` | -| done-marker | atomic write/read of `.status` | `harness/src/done-marker.ts` | -| job entrypoint | thin `main`: real queue + backend → `leaf-job-runner` | `knative-server/src/leaf-job.ts` | -| KEDA `ScaledJob` | redis-streams trigger → Job per pending entry, scale-to-zero, cap | `deploy/knative/leaf-scaledjob.yaml` | +| Unit | Responsibility | Lives in | +| ----------------- | -------------------------------------------------------------------------------- | ---------------------------------------- | +| enqueue handler | validate envelope, `XADD` to `leaf-queue`, return `202` + handle | `knative-server/src/server.ts` | +| status handler | report `queued\|running\|done\|failed` from the marker (+ queue state) | `knative-server/src/server.ts` | +| `WorkQueue` | Redis Streams primitive: `ensureGroup`/`enqueue`/`claim`/`ack`/`touch`/`pending` | `packages/work-queue` (`@sh/work-queue`) | +| `leaf-job-runner` | wrapper loop: claim → `runLeaf` → classify → marker → ack/reclaim | `harness/src/leaf-job-runner.ts` | +| `classifyOutcome` | pure ack-vs-reclaim decision (see §6) | `harness/src/classify-outcome.ts` | +| done-marker | atomic write/read of `.status` | `harness/src/done-marker.ts` | +| job entrypoint | thin `main`: real queue + backend → `leaf-job-runner` | `knative-server/src/leaf-job.ts` | +| KEDA `ScaledJob` | redis-streams trigger → Job per pending entry, scale-to-zero, cap | `deploy/knative/leaf-scaledjob.yaml` | **Key properties:** true background (Jobs outlive the request); scale-to-zero (no always-on worker); **at-least-once** delivery, where a crashed leaf re-runs the same `sessionId` → **gate-7 resume**; the @@ -133,7 +133,7 @@ A small JSON file the leaf-job writes **last, atomically** (temp file + rename): "ts": "" } ``` -Written *after* `result_ref` on success, or *instead of* it on failure — so a single file +Written _after_ `result_ref` on success, or _instead of_ it on failure — so a single file unambiguously signals terminal state for both outcomes (`result_ref` alone cannot: failures write none, and a result mid-write looks "present"). @@ -142,7 +142,7 @@ none, and a result mid-write looks "present"). - **Primary (canonical):** the orchestrator polls the **done-marker on its own volume**; the harness is not on the completion critical path. - **Secondary (convenience):** `GET /runs/status?sessionId=…` → `{ status: queued | running | - done | failed, reason? }`, where `done|failed` come from the marker and `queued|running` from the +done | failed, reason? }`, where `done|failed` come from the marker and `queued|running` from the queue/consumer-group state. ### 3.5 Idempotency & delivery semantics @@ -165,7 +165,7 @@ the stream is `leaf-queue:` (§7). ### 4.2 Delivery lifecycle (at-least-once) -- `XADD` → entry *pending* (unconsumed). +- `XADD` → entry _pending_ (unconsumed). - A leaf-job claims via `XREADGROUP … COUNT 1` → entry enters the group's **PEL** (delivered-but-unacked). - Terminal completion (`done` or a deterministic `failed`) → write `result_ref`/marker → **`XACK`** → @@ -236,12 +236,12 @@ from env). **`classifyOutcome` — the ack-vs-reclaim crux:** -| Outcome | done-marker | Queue action | Rationale | -|---|---|---|---| -| `runLeaf` → `done` | `done` | **XACK** | success; `result_ref` written | -| `runLeaf` → `failed: bad_inputs \| no_verdict \| invalid_verdict` | `failed` + reason | **XACK** | deterministic; re-running won't help (orchestrator re-dispatches with a *new* sessionId for a true retry) | -| `runLeaf` → `failed: error` (model/gateway blip) | none yet | **no ack → reclaim** | possibly transient; bounded retry via delivery count → on exhaustion, `failed` marker + ack | -| process **crash** (OOM/evict/SIGKILL, no return) | none | **no ack → reclaim** | entry stays in PEL → `XAUTOCLAIM` after `min-idle` → **gate-7 resume**; bounded by `maxAttempts` | +| Outcome | done-marker | Queue action | Rationale | +| ----------------------------------------------------------------- | ----------------- | -------------------- | --------------------------------------------------------------------------------------------------------- | +| `runLeaf` → `done` | `done` | **XACK** | success; `result_ref` written | +| `runLeaf` → `failed: bad_inputs \| no_verdict \| invalid_verdict` | `failed` + reason | **XACK** | deterministic; re-running won't help (orchestrator re-dispatches with a _new_ sessionId for a true retry) | +| `runLeaf` → `failed: error` (model/gateway blip) | none yet | **no ack → reclaim** | possibly transient; bounded retry via delivery count → on exhaustion, `failed` marker + ack | +| process **crash** (OOM/evict/SIGKILL, no return) | none | **no ack → reclaim** | entry stays in PEL → `XAUTOCLAIM` after `min-idle` → **gate-7 resume**; bounded by `maxAttempts` | **Effectively-once outcome:** at-least-once delivery × idempotent `runLeaf` (resume + overwrite) = the same verdict regardless of redelivery. A rare double-process (partition after the heartbeat @@ -254,14 +254,14 @@ noted as an honest caveat. ### 6.1 Unit (fast, vitest) -| Unit | Coverage | -|---|---| -| `WorkQueue` | `XADD`→`XREADGROUP` returns the entry; `XACK` clears the PEL; `XAUTOCLAIM` reclaims past `min-idle`; delivery-count increments; `pending`. Against a real Redis, **gated** like the existing `redis-backend` tests (skip when no `REDIS_URL`). | -| `classifyOutcome` | pure §5 table: `done`→ack+done; `bad_inputs/no_verdict/invalid_verdict`→ack+failed; `error`→no-ack/retryable; exhausted→failed+ack. | -| enqueue handler | `vi.mock` queue → `202`+`XADD` once on valid async envelope; `400` no-`XADD` on malformed; **async omitted/false still runs the sync path** (no regression). | -| status handler | marker present→done/failed; absent+pending→queued/running (injected marker-reader + queue-state). | -| `leaf-job-runner` | injected fakes (queue, `runLeaf`, marker writer, clock): autoclaim-preferred-over-read; dead-letter when delivery>max; heartbeat scheduled+cleared; ack-on-terminal; no-ack+rethrow on retryable. | -| done-marker | atomic temp+rename; correct JSON for `done` and `failed`. | +| Unit | Coverage | +| ----------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `WorkQueue` | `XADD`→`XREADGROUP` returns the entry; `XACK` clears the PEL; `XAUTOCLAIM` reclaims past `min-idle`; delivery-count increments; `pending`. Against a real Redis, **gated** like the existing `redis-backend` tests (skip when no `REDIS_URL`). | +| `classifyOutcome` | pure §5 table: `done`→ack+done; `bad_inputs/no_verdict/invalid_verdict`→ack+failed; `error`→no-ack/retryable; exhausted→failed+ack. | +| enqueue handler | `vi.mock` queue → `202`+`XADD` once on valid async envelope; `400` no-`XADD` on malformed; **async omitted/false still runs the sync path** (no regression). | +| status handler | marker present→done/failed; absent+pending→queued/running (injected marker-reader + queue-state). | +| `leaf-job-runner` | injected fakes (queue, `runLeaf`, marker writer, clock): autoclaim-preferred-over-read; dead-letter when delivery>max; heartbeat scheduled+cleared; ack-on-terminal; no-ack+rethrow on retryable. | +| done-marker | atomic temp+rename; correct JSON for `done` and `failed`. | ### 6.2 Live gate — `deploy/knative/leaf-async-smoke.sh` (gated `ASYNC_LIVE_SMOKE=1`) @@ -289,7 +289,7 @@ The queue substrate adapts to **per-user queues** without redesign: an optional namespaces the stream (`leaf-queue:` + its own consumer group) and the `sessionId` (prefixed, then `toSessionId`-sanitized). This yields separate backlogs, fairness, and scaling per tenant. -**KEDA caveat:** a `ScaledJob` watches *one* stream. A small, known tenant set → one `ScaledJob` per +**KEDA caveat:** a `ScaledJob` watches _one_ stream. A small, known tenant set → one `ScaledJob` per tenant (true per-user queues). Dynamic/many users → either provision a `ScaledJob` per tenant at onboarding, or run a single `ScaledJob` over a shared stream with tenant-tagged entries + app-level per-tenant caps (workload separation, not separate queues). @@ -316,7 +316,7 @@ stream-per-tenant + `ScaledJob`-per-tenant wiring is **not** built. - A separate dead-letter queue stream (dead-letter = `failed` marker + ack). - Changes to synchronous `POST /runs` (unchanged). -**Substrate-agnostic seam:** the async *contract* (enqueue envelope → background execution → +**Substrate-agnostic seam:** the async _contract_ (enqueue envelope → background execution → `result_ref` + done-marker + status) depends only on the `WorkQueue` interface + `leaf-job-runner`, not KEDA. A KEDA-less cluster could drive the same queue with harness-created Jobs or an always-on worker without changing `runLeaf` / done-marker / status — the documented fallback. @@ -340,4 +340,4 @@ worker without changing `runLeaf` / done-marker / status — the documented fall --- -*Assisted-By: Claude (Anthropic AI) * +_Assisted-By: Claude (Anthropic AI) _ diff --git a/docs/specs/2026-06-28-human-gate-design.md b/docs/specs/2026-06-28-human-gate-design.md index 7d89699..be843d8 100644 --- a/docs/specs/2026-06-28-human-gate-design.md +++ b/docs/specs/2026-06-28-human-gate-design.md @@ -18,7 +18,7 @@ multi-tenancy & per-user identity (Z1), credential plane (Z3/Z5), event-driven g > **One-line finding.** The gate is not new machinery — it is **a structured-output terminal plus a > decision-seeded continuation**, both of which the substrate already does. A leaf reaching a gate -> ends a turn *well-formed* and parks (its session log is durable, the pod scales to zero); an +> ends a turn _well-formed_ and parks (its session log is durable, the pod scales to zero); an > external approver writes a decision file and re-invokes the same `sessionId`; `runLeaf` resumes > from the log, applies the decision, and continues with full context. No always-on component, no Pi > internals dependency, no change to any existing path. @@ -61,13 +61,13 @@ multi-tenancy & per-user identity (Z1), credential plane (Z3/Z5), event-driven g ### Design decisions (resolved with the stakeholder) -| # | Decision | Choice | Rationale | -|---|----------|--------|-----------| -| B1 | Who decides **where** a gate happens | **Agent-declared** via a tool; **decision + resume external** (charter-aligned "Option C") | Builds the novel single-session pause/resume primitive while decision authority and re-invocation stay external (G1/G2). | -| B2 | How the agent **continues** after a decision | **Continuation prompt** (not tool-result injection) | Turn ends well-formed (no dangling tool call); reuses the proven seed-prompt + M5 resume path; survives mid-park crashes; no Pi-internals dependency. | -| B3 | **Decision transport** + resume trigger | **`decisionRef` file** on the volume; resume = **re-invoke `POST /runs` by `sessionId`** (sync or `async:true`) | Symmetric with `inputsRef`/`resultRef`; keeps decision content off the HTTP channel (G3); no harness-side watcher (preserves scale-to-zero + G1/G2). | -| B4 | **Timeout** of a parked gate | **None in the harness**; park indefinitely | Charter §8.3 ("sleep indefinitely"); a timer/sweeper would reintroduce an always-on component. Deadlines are the orchestrator's concern (re-invoke with `abort`). | -| B5 | **Decision actions** | `approve` \| `reject` \| `abort` | Matches archetype B's approve / reject→loop / abort. | +| # | Decision | Choice | Rationale | +| --- | -------------------------------------------- | --------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| B1 | Who decides **where** a gate happens | **Agent-declared** via a tool; **decision + resume external** (charter-aligned "Option C") | Builds the novel single-session pause/resume primitive while decision authority and re-invocation stay external (G1/G2). | +| B2 | How the agent **continues** after a decision | **Continuation prompt** (not tool-result injection) | Turn ends well-formed (no dangling tool call); reuses the proven seed-prompt + M5 resume path; survives mid-park crashes; no Pi-internals dependency. | +| B3 | **Decision transport** + resume trigger | **`decisionRef` file** on the volume; resume = **re-invoke `POST /runs` by `sessionId`** (sync or `async:true`) | Symmetric with `inputsRef`/`resultRef`; keeps decision content off the HTTP channel (G3); no harness-side watcher (preserves scale-to-zero + G1/G2). | +| B4 | **Timeout** of a parked gate | **None in the harness**; park indefinitely | Charter §8.3 ("sleep indefinitely"); a timer/sweeper would reintroduce an always-on component. Deadlines are the orchestrator's concern (re-invoke with `abort`). | +| B5 | **Decision actions** | `approve` \| `reject` \| `abort` | Matches archetype B's approve / reject→loop / abort. | --- @@ -88,8 +88,8 @@ On call the harness: agent can retry, exactly like `submit_verdict`). 2. Appends a durable **gate-request** custom entry: `{ gateId, summary, proposed_action }`, where `gateId` = the count of prior gate-request entries in this session (0, 1, 2, …). -3. Returns a **benign synchronous result**: *"Approval requested; the session will pause and resume - with the human decision."* — so the assistant turn ends **well-formed** (no dangling `tool_use`; +3. Returns a **benign synchronous result**: _"Approval requested; the session will pause and resume + with the human decision."_ — so the assistant turn ends **well-formed** (no dangling `tool_use`; this is what makes resume robust, B2). The tool sets a capture flag (`gateRequested`, with the new `gateId`) that `runLeaf` inspects after @@ -211,7 +211,7 @@ A front-end runs on **every** invocation (fresh or resume), before the agent loo ``` **`gateId` is the idempotency spine.** A session may pass several gates; the marker advertises the -*current* `gateId` and the decision file must echo it. A resume applies a decision **only** when its +_current_ `gateId` and the decision file must echo it. A resume applies a decision **only** when its `gateId` matches the pending gate, so: - a **stale/replayed** decision (answering an already-consumed gate) is **ignored** (branch (b)); @@ -238,29 +238,29 @@ flushed through `BufferedRedisBackend`. If the pod crashes after the decision is a verdict, the reclaim/re-invoke finds the gate already decided (no pending gate), takes branch (c), **re-derives the same continuation prompt** from the durable gate-decision entry, and re-runs — the decision is never recorded twice and the re-run is idempotent (overwrites to the same verdict). The -gate-decision entry is the idempotency guard that prevents a *second* decision from being recorded +gate-decision entry is the idempotency guard that prevents a _second_ decision from being recorded for a consumed `gateId`. --- ## 4. Async integration (`classifyOutcome` + KEDA) -**Two marker kinds, two writers.** The async substrate already has a *terminal* done-marker at +**Two marker kinds, two writers.** The async substrate already has a _terminal_ done-marker at `.status` written by the **queue runner** (via `classifyOutcome.marker`), absent in sync -mode (the HTTP response conveys the terminal status). The gate adds a distinct *non-terminal* **gate +mode (the HTTP response conveys the terminal status). The gate adds a distinct _non-terminal_ **gate marker** at `gateRef` (`.gate`) that carries the session-derived summary — so **`runLeaf` writes it directly, in both sync and async modes** (the queue runner only has the `LeafResult`, not the summary). `classifyOutcome` therefore returns **no terminal marker** for `paused` (runLeaf already wrote the gate marker); it only decides the ack: -| `runLeaf` outcome | gate marker (`gateRef`, by `runLeaf`) | terminal marker (`.status`, by runner/async) | queue action | rationale | -|---|---|---|---|---| -| `done` | — | `done` | **XACK** | success (unchanged) | -| **`paused`** | **`awaiting_approval`** (written by `runLeaf`) | — (none; not terminal) | **XACK** | parked; resume is a *new* invocation, not a redelivery | -| **`aborted`** | — | **`aborted`** | **XACK** | terminal by human decision | -| `failed: bad_inputs \| no_verdict \| invalid_verdict` | — | `failed` | **XACK** | deterministic (unchanged) | -| `failed: error` | — | none | **no ack → reclaim** | transient (unchanged) | -| process **crash** | — | none | **no ack → reclaim → gate-7 resume** | unchanged | +| `runLeaf` outcome | gate marker (`gateRef`, by `runLeaf`) | terminal marker (`.status`, by runner/async) | queue action | rationale | +| ----------------------------------------------------- | ---------------------------------------------- | ------------------------------------------------------- | ------------------------------------ | ------------------------------------------------------ | +| `done` | — | `done` | **XACK** | success (unchanged) | +| **`paused`** | **`awaiting_approval`** (written by `runLeaf`) | — (none; not terminal) | **XACK** | parked; resume is a _new_ invocation, not a redelivery | +| **`aborted`** | — | **`aborted`** | **XACK** | terminal by human decision | +| `failed: bad_inputs \| no_verdict \| invalid_verdict` | — | `failed` | **XACK** | deterministic (unchanged) | +| `failed: error` | — | none | **no ack → reclaim** | transient (unchanged) | +| process **crash** | — | none | **no ack → reclaim → gate-7 resume** | unchanged | So `classifyOutcome`'s new branches are: `paused` → `{ ack: true, marker: null }`; `aborted` → `{ ack: true, marker: { status: "aborted" } }`. The resume invocation is an ordinary sync POST or @@ -275,7 +275,7 @@ async enqueue with the same `sessionId` + `decisionRef`. On the async path it `X > secondary convenience path (async §3.4). > **Marker overwrite across gates.** A second gate in the same session overwrites the -> `awaiting_approval` gate marker with the new `gateId`. The orchestrator detects a *new* gate by the +> `awaiting_approval` gate marker with the new `gateId`. The orchestrator detects a _new_ gate by the > changed `gateId` (and, if it wants history, can snapshot markers — its concern, G3). --- @@ -298,8 +298,8 @@ async enqueue with the same `sessionId` + `decisionRef`. On the async path it `X - **`decisionRef` missing/garbled on resume:** treated as "no decision" → branch (b) → stays `paused` (no agent run). Safe no-op; the approver re-writes and re-invokes. - **Timeout:** **none in the harness** (B4). A parked session waits indefinitely; the orchestrator - enforces deadlines by re-invoking with `abort`. *(Non-precluding future: an envelope `gateTTL` + - a KEDA-cron sweeper that writes `aborted` markers — not built; see §7.)* + enforces deadlines by re-invoking with `abort`. _(Non-precluding future: an envelope `gateTTL` + + a KEDA-cron sweeper that writes `aborted` markers — not built; see §7.)_ - **Re-entrancy preserved:** the gate adds no caller-identity assumption, so a child leaf could itself gate. Non-precluding, untested (consistent with MVP §2.5). @@ -311,19 +311,19 @@ double-resume. ## 6. Components -| # | Unit | Responsibility | Lives in | New / reuse | -|---|---|---|---|---| -| 1 | `request_approval` tool | validate → append gate-request entry → benign result; set capture flag | `harness/src/request-approval-tool.ts` | ⭐ new | -| 2 | gate types + validation | `GateRequest`, `Decision`, `validateDecision`, `gateId` derivation | `harness/src/gate.ts` | ⭐ new | -| 3 | gate marker I/O | atomic write/read of the `awaiting_approval` gate marker; `deriveGateRef` (`.gate`) | `harness/src/gate-marker.ts` (mirrors `done-marker.ts`) | ⭐ new | -| 4 | resume state machine | pending-gate detection, decision application, continuation seeding, abort, **gate-marker write on park** | extend `harness/src/run-leaf.ts` | ⭐ new front-end; loop reused | -| 5 | `LeafResult` + envelope additions | `paused`/`aborted`; `gateRef`/`decisionRef` | extend `harness/src/run-leaf.ts` | ⭐ new fields | -| 6 | `classifyOutcome` | `paused`→`{ack:true, marker:null}` (gate marker already written by `runLeaf`); `aborted`→`{ack:true, marker:aborted}` | extend `harness/src/classify-outcome.ts` | ⭐ new branches | -| 7 | server handlers | serialize `paused`/`aborted` on sync `200`; `decisionRef`/`gateRef` through `isLeafEnvelope`; status endpoint reports `awaiting_approval` from the gate marker | `packages/knative-server/src/server.ts` | ♻️ minimal | -| 8 | leaf-job runner | unchanged loop; new outcomes ack via `classifyOutcome` | `harness/src/leaf-job-runner.ts` | ♻️ unchanged | -| 9 | session durability / resume | gate entries through `BufferedRedisBackend`; `openFromCheckpoint` | M1/M5 | ♻️ reuse | -| 10 | live gate smoke | `GATE_LIVE_SMOKE=1` end-to-end on Kind | `deploy/knative/leaf-gate-smoke.sh` | ⭐ new | -| 11 | fixture gating prompt | a prompt that requests approval once before verdict | test fixtures | ⭐ new (small) | +| # | Unit | Responsibility | Lives in | New / reuse | +| --- | --------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------- | ----------------------------- | +| 1 | `request_approval` tool | validate → append gate-request entry → benign result; set capture flag | `harness/src/request-approval-tool.ts` | ⭐ new | +| 2 | gate types + validation | `GateRequest`, `Decision`, `validateDecision`, `gateId` derivation | `harness/src/gate.ts` | ⭐ new | +| 3 | gate marker I/O | atomic write/read of the `awaiting_approval` gate marker; `deriveGateRef` (`.gate`) | `harness/src/gate-marker.ts` (mirrors `done-marker.ts`) | ⭐ new | +| 4 | resume state machine | pending-gate detection, decision application, continuation seeding, abort, **gate-marker write on park** | extend `harness/src/run-leaf.ts` | ⭐ new front-end; loop reused | +| 5 | `LeafResult` + envelope additions | `paused`/`aborted`; `gateRef`/`decisionRef` | extend `harness/src/run-leaf.ts` | ⭐ new fields | +| 6 | `classifyOutcome` | `paused`→`{ack:true, marker:null}` (gate marker already written by `runLeaf`); `aborted`→`{ack:true, marker:aborted}` | extend `harness/src/classify-outcome.ts` | ⭐ new branches | +| 7 | server handlers | serialize `paused`/`aborted` on sync `200`; `decisionRef`/`gateRef` through `isLeafEnvelope`; status endpoint reports `awaiting_approval` from the gate marker | `packages/knative-server/src/server.ts` | ♻️ minimal | +| 8 | leaf-job runner | unchanged loop; new outcomes ack via `classifyOutcome` | `harness/src/leaf-job-runner.ts` | ♻️ unchanged | +| 9 | session durability / resume | gate entries through `BufferedRedisBackend`; `openFromCheckpoint` | M1/M5 | ♻️ reuse | +| 10 | live gate smoke | `GATE_LIVE_SMOKE=1` end-to-end on Kind | `deploy/knative/leaf-gate-smoke.sh` | ⭐ new | +| 11 | fixture gating prompt | a prompt that requests approval once before verdict | test fixtures | ⭐ new (small) | --- @@ -331,15 +331,15 @@ double-resume. ### 7.1 Unit (vitest, fast — pure / injected fakes) -| Unit | Coverage | -|---|---| -| `request_approval` tool | valid args → gate-request entry with correct `gateId` + benign result; invalid (empty `summary`/`proposed_action`) → tool error, no entry. | -| gate types | `validateDecision` accepts `approve`/`reject`/`abort` + optional feedback; rejects bad action / missing `gateId`. | -| gate marker I/O | atomic temp+rename for the `awaiting_approval` gate marker; `deriveGateRef` default (`.gate`) + override. | +| Unit | Coverage | +| ---------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `request_approval` tool | valid args → gate-request entry with correct `gateId` + benign result; invalid (empty `summary`/`proposed_action`) → tool error, no entry. | +| gate types | `validateDecision` accepts `approve`/`reject`/`abort` + optional feedback; rejects bad action / missing `gateId`. | +| gate marker I/O | atomic temp+rename for the `awaiting_approval` gate marker; `deriveGateRef` default (`.gate`) + override. | | resume state machine (injected store / loop / clock) | pending-gate detection; **approve**→durable gate-decision entry + continuation seed + run + gate-marker on a follow-on gate; **reject**→feedback seed + run; **abort**→aborted (terminal), **no** agent run; **gateId mismatch**→ignored (stays paused); **duplicate decision**→no-op continuation (no second seed); **no decisionRef**→paused; **fresh**→initial seed. | -| `classifyOutcome` | `paused`→`{ack:true, marker:null}`; `aborted`→`{ack:true, marker:aborted}`; existing rows unchanged (regression). | -| multi-gate sequence | gate#0 → approve → gate#1 → approve → verdict, single `sessionId`, correct `gateId` progression. | -| server | `paused`/`aborted` serialize correctly on sync `200`; `decisionRef` passes through `isLeafEnvelope`; async-omitted path unchanged (regression). | +| `classifyOutcome` | `paused`→`{ack:true, marker:null}`; `aborted`→`{ack:true, marker:aborted}`; existing rows unchanged (regression). | +| multi-gate sequence | gate#0 → approve → gate#1 → approve → verdict, single `sessionId`, correct `gateId` progression. | +| server | `paused`/`aborted` serialize correctly on sync `200`; `decisionRef` passes through `isLeafEnvelope`; async-omitted path unchanged (regression). | Redis-backed assertions are **gated** like existing `redis-backend` tests (skip when no `REDIS_URL`). @@ -350,7 +350,7 @@ On the Kind `sh-knative` cluster (async path already deployed: KEDA + `ScaledJob **The controller runs this live gate directly — never a subagent.** 1. **Pause:** dispatch a gated leaf → `awaiting_approval` marker appears on `/work` with `gateId:0` - + summary; **no** `result_ref`; session parked. + - summary; **no** `result_ref`; session parked. 2. **Scale-to-zero while parked:** with the gate pending and no other work, leaf-job pods reach **zero** (KEDA acked the parked entry). 3. **Resume-approve:** write `decisionRef {gateId:0, approve}` → re-invoke same `sessionId` → leaf @@ -378,7 +378,7 @@ On the Kind `sh-knative` cluster (async path already deployed: KEDA + `ScaledJob `ScaledJob`, `submit_verdict`, or the heartbeat/dead-letter machinery — all **unchanged**. - A new container image — the gate reuses the harness image and existing entrypoints. -**Substrate-agnostic seam:** the gate *contract* (gate tool → `awaiting_approval` marker → external +**Substrate-agnostic seam:** the gate _contract_ (gate tool → `awaiting_approval` marker → external `decisionRef` → re-invoke-by-`sessionId` → continuation) depends only on `runLeaf`'s durable session log + the marker convention, **not** on KEDA or Knative. A KEDA-less or always-on deployment drives the identical contract. @@ -404,4 +404,4 @@ the identical contract. --- -*Assisted-By: Claude (Anthropic AI) * +_Assisted-By: Claude (Anthropic AI) _ diff --git a/docs/specs/2026-06-28-registry-hardening-hygiene-design.md b/docs/specs/2026-06-28-registry-hardening-hygiene-design.md index 90773e6..9731ea1 100644 --- a/docs/specs/2026-06-28-registry-hardening-hygiene-design.md +++ b/docs/specs/2026-06-28-registry-hardening-hygiene-design.md @@ -27,12 +27,12 @@ recorded in any table. The roadmap is stale w.r.t. what merged. **Change.** Add a new section **"Leaf-Session Backend (BUILT)"** between the Phase-1 and Phase-2 sections, with a table: -| Slice | Spec | PR | -|---|---|---| -| MVP leaf-session invocation contract | `2026-06-26-mvp-leaf-session-contract-design.md` | #10, #11 (gate-7 resume) | -| Async leaf completion (KEDA `ScaledJob` + queue) | `2026-06-27-async-leaf-completion-design.md` | #12 | -| Scheduled leaf dispatch (cron trigger on-ramp) | `2026-06-28-scheduled-leaf-dispatch-design.md` | #13 | -| Human-gate (gate-while-idle, Archetype B) | `2026-06-28-human-gate-design.md` | #14 | +| Slice | Spec | PR | +| ------------------------------------------------ | ------------------------------------------------ | ------------------------ | +| MVP leaf-session invocation contract | `2026-06-26-mvp-leaf-session-contract-design.md` | #10, #11 (gate-7 resume) | +| Async leaf completion (KEDA `ScaledJob` + queue) | `2026-06-27-async-leaf-completion-design.md` | #12 | +| Scheduled leaf dispatch (cron trigger on-ramp) | `2026-06-28-scheduled-leaf-dispatch-design.md` | #13 | +| Human-gate (gate-while-idle, Archetype B) | `2026-06-28-human-gate-design.md` | #14 | A short paragraph notes: these realize the [Capability Charter](2026-06-26-leaf-session-backend-capability-charter.md) §5 MVP core + §8 promote-post-MVP (human-gate, cron trigger), sit **outside** the `M`/`Z` numbering @@ -96,9 +96,10 @@ reason inline. The non-fs hardening (non-root, runAsUser, seccomp, no-priv-esc, applied unconditionally regardless. **Risks this surfaces (all caught by the live smoke):** + - `runAsUser: 65532` must be able to **write the `/work` PVC** (the gate writes `result_ref`/markers there). `fsGroup: 65532` makes the harness's own writes group-owned, **but** `fsGroup` does NOT - make a *result directory created by a different (root) writer* group-writable. **Operational + make a _result directory created by a different (root) writer_ group-writable. **Operational contract (confirmed in live verification — EACCES writing the gate marker):** because `/work` is the orchestrator's store (charter G3) and the harness now runs as uid 65532, the **orchestrator must provision the per-run result directories writable by uid 65532** (e.g. world-writable, or @@ -148,4 +149,4 @@ applied unconditionally regardless. --- -*Assisted-By: Claude (Anthropic AI) * +_Assisted-By: Claude (Anthropic AI) _ diff --git a/docs/specs/2026-06-28-scheduled-leaf-dispatch-design.md b/docs/specs/2026-06-28-scheduled-leaf-dispatch-design.md index 352aeb2..8fbf7bf 100644 --- a/docs/specs/2026-06-28-scheduled-leaf-dispatch-design.md +++ b/docs/specs/2026-06-28-scheduled-leaf-dispatch-design.md @@ -20,8 +20,8 @@ contract + `runLeaf` (MVP/PR #10), gate-7 resume (PR #11). ## 1. Goal & motivation -Archetypes A (parallel fan-out) and B (iterative loop) start from an *invocation* — an external -orchestrator dispatches work. **Archetype C — scheduled ingestion** starts from a *clock*: "every +Archetypes A (parallel fan-out) and B (iterative loop) start from an _invocation_ — an external +orchestrator dispatches work. **Archetype C — scheduled ingestion** starts from a _clock_: "every night at 02:00, process the standing batch." The harness already accepts background work via `POST /runs {async:true}`; this slice adds the missing **start signal** so a schedule — with no external orchestrator process running — can dispatch that work. @@ -32,11 +32,11 @@ post-MVP. This realizes the **cron** half of it. ### Charter fit (why this respects "harness is invoked, not orchestrator") -The harness charter (G1/G2) is that the harness is *invoked*, never the orchestrator. A cron schedule +The harness charter (G1/G2) is that the harness is _invoked_, never the orchestrator. A cron schedule that dispatches a **fixed, config-defined list** does not violate this: the schedule and the list are **operator-supplied configuration**, not a decision the harness computes. The CronJob is a thin -*client* of the unchanged async contract — equivalent to an external caller that happens to be driven -by a clock. Dynamic work-selection (deciding *what* to process at fire time) would be orchestration +_client_ of the unchanged async contract — equivalent to an external caller that happens to be driven +by a clock. Dynamic work-selection (deciding _what_ to process at fire time) would be orchestration and is explicitly out of scope (§8). --- @@ -65,11 +65,11 @@ operator/orchestrator reads markers under the fire-stamped resultRef dir **Components** (each independently testable): -| Unit | Responsibility | Lives in | -|---|---|---| -| `cron-dispatch` | read config + fire id → POST each templated envelope async; aggregate result → exit code | `packages/knative-server/src/cron-dispatch.ts` | -| schedule + item list | **one manifest, two YAML docs:** a ConfigMap (envelope list + `sessionId`/`resultRef` templates with `__FIRE__`) and the CronJob (schedule, `concurrencyPolicy: Forbid`, downward-API Job-name env, ConfigMap mount, harness image, `serverless-harness` SA). Single `kubectl apply -f`. | `deploy/knative/leaf-cron.yaml` | -| live gate | gated smoke: a fire dispatches the list → leaves complete; dispatcher-retry is idempotent | `deploy/knative/leaf-cron-smoke.sh` | +| Unit | Responsibility | Lives in | +| -------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------- | +| `cron-dispatch` | read config + fire id → POST each templated envelope async; aggregate result → exit code | `packages/knative-server/src/cron-dispatch.ts` | +| schedule + item list | **one manifest, two YAML docs:** a ConfigMap (envelope list + `sessionId`/`resultRef` templates with `__FIRE__`) and the CronJob (schedule, `concurrencyPolicy: Forbid`, downward-API Job-name env, ConfigMap mount, harness image, `serverless-harness` SA). Single `kubectl apply -f`. | `deploy/knative/leaf-cron.yaml` | +| live gate | gated smoke: a fire dispatches the list → leaves complete; dispatcher-retry is idempotent | `deploy/knative/leaf-cron-smoke.sh` | **Key properties:** no new harness logic on the enqueue path (the dispatcher is a client of the existing contract); no Job RBAC beyond the existing `serverless-harness` ServiceAccount; the schedule @@ -88,16 +88,24 @@ its cadence; `kubectl create job --from=cronjob/leaf-cron ` to fire one on schedule-submission API — registering schedules is the operator's `kubectl` (or the external orchestrator's), not a harness control plane (charter G1/G2). - ```json -{ "items": [ - { "sessionId": "nightly/__FIRE__/i1", - "inputsRef": "/work/nightly/inputs/i1.json", - "resultRef": "/work/nightly/__FIRE__/results/i1.json", - "workspaceRef":"/workspace/nightly/repo", - "model": "claude-haiku-4-5" }, - { "sessionId": "nightly/__FIRE__/i2", "inputsRef": "…/i2.json", "resultRef": "/work/nightly/__FIRE__/results/i2.json", "workspaceRef": "/workspace/nightly/repo" } -] } +{ + "items": [ + { + "sessionId": "nightly/__FIRE__/i1", + "inputsRef": "/work/nightly/inputs/i1.json", + "resultRef": "/work/nightly/__FIRE__/results/i1.json", + "workspaceRef": "/workspace/nightly/repo", + "model": "claude-haiku-4-5" + }, + { + "sessionId": "nightly/__FIRE__/i2", + "inputsRef": "…/i2.json", + "resultRef": "/work/nightly/__FIRE__/results/i2.json", + "workspaceRef": "/workspace/nightly/repo" + } + ] +} ``` - `__FIRE__` is the only template token; the dispatcher replaces it (every occurrence, in every @@ -113,7 +121,7 @@ The **fire id** is the dispatcher pod's owning **Job name**, read via the downwa - **Unique per scheduled fire:** the CronJob names each Job `…-`. - **Stable across a Job's pod retries:** if the dispatcher pod fails and `backoffLimit` restarts it, - the new pod is under the *same* Job → same fire id → re-POSTs the *same* `sessionId`s. + the new pod is under the _same_ Job → same fire id → re-POSTs the _same_ `sessionId`s. ### 3.3 Idempotency & delivery semantics @@ -124,7 +132,7 @@ The **fire id** is the dispatcher pod's owning **Job name**, read via the downwa **resumes/overwrites** rather than duplicating — at-least-once dispatch with effectively-once outcome, consistent with [async §3.5](2026-06-27-async-leaf-completion-design.md) and MVP §2.4. - `concurrencyPolicy: Forbid` prevents two dispatcher Jobs for the same CronJob overlapping (a slow - fire won't be double-started by the next tick). Leaf executions across *different* fires may + fire won't be double-started by the next tick). Leaf executions across _different_ fires may overlap freely — they are distinct `sessionId`s. ### 3.4 Dispatcher result → exit code @@ -141,12 +149,12 @@ malformed envelope, which the dispatcher surfaces as a failed fire. ## 4. Why a CronJob, not the KEDA cron scaler The async design noted "KEDA's cron scaler is the on-ramp." On implementation that is the wrong -primitive: KEDA's `cron` scaler is **window-based** — it scales a workload's replicas *up between* a +primitive: KEDA's `cron` scaler is **window-based** — it scales a workload's replicas _up between_ a `start`/`end` cron pair and back down after, intended for "keep N replicas warm during business hours." It does not model a **discrete fire at time T**, and on a `ScaledJob` it would spawn jobs continuously across the active window. The native Kubernetes **`CronJob`** is the correct discrete -scheduler. KEDA remains the right tool for the *queue* half (already in use for the `ScaledJob`); the -*schedule* half is a `CronJob`. This supersedes the async spec's passing note. +scheduler. KEDA remains the right tool for the _queue_ half (already in use for the `ScaledJob`); the +_schedule_ half is a `CronJob`. This supersedes the async spec's passing note. --- @@ -170,6 +178,7 @@ The dispatcher POSTs to the Knative service over cluster-internal networking: ### 6.1 Unit (vitest, pure — injected `fetch`, no cluster) `cron-dispatch`: + - each config item is POSTed exactly once to `/runs` with `async:true`; - `__FIRE__` is substituted with the fire id in `sessionId` **and** `resultRef` (and any other field containing it); non-templated fields pass through verbatim; @@ -232,4 +241,4 @@ out here so the harness performs no work-selection. --- -*Assisted-By: Claude (Anthropic AI) * +_Assisted-By: Claude (Anthropic AI) _ diff --git a/docs/specs/2026-07-02-p0prime-ocp-fs-free-deployment-design.md b/docs/specs/2026-07-02-p0prime-ocp-fs-free-deployment-design.md index ee37f98..6d260fd 100644 --- a/docs/specs/2026-07-02-p0prime-ocp-fs-free-deployment-design.md +++ b/docs/specs/2026-07-02-p0prime-ocp-fs-free-deployment-design.md @@ -52,6 +52,7 @@ sandbox will too — §5.2). No `anyuid`, no privileged pods. Verified against `main` @ `d199484` and a read-only inspection of the live cluster. ### 3.1 Live cluster (ready substrate) + - OCP **4.20.8** / k8s 1.33.6; kubeconfig authenticates as cluster-admin. - OpenShift Serverless **KnativeServing v1.17 Ready**; KEDA Ready (`openshift-keda`). - StorageClasses `gp3-csi` (default) + `gp2-csi` — both AWS EBS, **RWO, `WaitForFirstConsumer`**, no RWX. @@ -60,6 +61,7 @@ Verified against `main` @ `d199484` and a read-only inspection of the live clust - No existing harness namespace — clean slate. ### 3.2 The four breakages (post-P1) + 1. **Stale resource reference.** `deploy/knative/overlays/ocp/kustomization.yaml` still lists the **deleted** `../../leaf-pvc.yaml` → `kustomize build` fails outright. (P1 removed `leaf-pvc.yaml`.) 2. **Wrong patch target.** `overlays/ocp/patch-sandbox.yaml` is `kind: Pod, name: sandbox-0`, but the @@ -71,12 +73,13 @@ Verified against `main` @ `d199484` and a read-only inspection of the live clust `volumeClaimTemplates` (RWO 1Gi) must back `/workspace`. `emptyDir` also silently breaks the smoke's crash/resume claim (a pod restart would wipe the seeded repo). 4. **Missing `ripgrep`.** `deploy/knative/sandbox.Dockerfile` installs `bash coreutils findutils - grep` but **not `ripgrep`**. The leaf's `find`/`grep` tools route to `rg` inside the sandbox +grep` but **not `ripgrep`**. The leaf's `find`/`grep` tools route to `rg` inside the sandbox (`packages/k8s-sandbox/src/operations.ts`, `grep-tool.ts`), so the full leaf smoke crashes on the first search. Kind avoids this because its base `sandbox.yaml` does `apk add … ripgrep` at startup; the OCP image pre-bakes tools (no root `apk add` under `restricted-v2`) and omitted `rg`. ### 3.3 `setup-ocp.sh` gaps (vs. the working `setup-kind.sh`) + - **Never installs the agent-sandbox controller** (Kind applies the v0.5.0 `manifest.yaml` + `kubectl wait` the CRD Established). - **Never applies/awaits the Sandbox CR** — no `.status.selector` poll → pod-Ready (Kind polls up to @@ -103,10 +106,11 @@ GHCR-pulled (`ghcr.io/rossoctl/serverless-harness:latest`, auto-published post-P ## 5. Design ### 5.1 Target topology (P1 architecture, realized on OCP) + - **Harness** — Knative Service, image `ghcr.io/rossoctl/serverless-harness:latest`, runs **non-root** (UID 65532, `nonroot-v2`), mounts only `/tmp` (emptyDir). Resolves the sandbox pod via `KAGENTI_SANDBOX_NAME=sandbox-0` → the `Sandbox` CR's `.status.selector` label query → `kubectl - exec`s all 7 Pi tool ops into it. External access via the Knative **Route** (`KSVC_URL` contract in +exec`s all 7 Pi tool ops into it. External access via the Knative **Route** (`KSVC_URL` contract in `lib.sh`: Route host, `-k`, no Host header). - **Sandbox** — one `Sandbox` CR (`sandbox-0`, `agents.x-k8s.io`), managed by the agent-sandbox **v0.5.0** controller. `/workspace` backed by a **durable RWO EBS PVC** from the CR's @@ -115,8 +119,10 @@ GHCR-pulled (`ghcr.io/rossoctl/serverless-harness:latest`, auto-published post-P - **Redis** — `deploy/knative/redis.yaml` (result record + async queue), unchanged. ### 5.2 Sandbox SCC / non-root model (the OCP-specific core) + Chosen approach: **non-root + `fsGroup`, bound under `nonroot-v2`.** Realized by patching the **`Sandbox` CR `podTemplate`** in `overlays/ocp/patch-sandbox.yaml`: + - Pod `securityContext`: `runAsUser: 65532`, `runAsNonRoot: true`, **`fsGroup: 65532`**, `seccompProfile.type: RuntimeDefault`. - Container `securityContext`: `allowPrivilegeEscalation: false`, `capabilities.drop: ["ALL"]`. @@ -124,7 +130,7 @@ Chosen approach: **non-root + `fsGroup`, bound under `nonroot-v2`.** Realized by granted `nonroot-v2` (`oc adm policy add-scc-to-user nonroot-v2 -z serverless-harness-sandbox …` in `setup-ocp.sh`). - **Keep** the durable PVC from `volumeClaimTemplates` (remove the `emptyDir` override). `fsGroup: - 65532` makes the EBS volume group-owned/writable, so the sandbox — and the smoke's `sexec` +65532` makes the EBS volume group-owned/writable, so the sandbox — and the smoke's `sexec` repo-seeding (`leaf-smoke.sh` writes `/workspace//repo` via `kubectl exec`) — writes as 65532. **Kustomize mechanism.** Patching a container inside a CRD's `podTemplate` via strategic-merge is a @@ -134,17 +140,19 @@ strategic-merge patch of `kind: Sandbox` (not `Pod`) if the field-level merge is The plan picks one after a quick render check; either way the patch is **onto the `Sandbox` CR**. ### 5.3 Concrete changeset (P1-slice only) -| File | Change | -|------|--------| -| `deploy/knative/sandbox.Dockerfile` | Add `ripgrep` to the `apk add` line. | -| `deploy/knative/overlays/ocp/kustomization.yaml` | Remove the `../../leaf-pvc.yaml` resource; keep the `images:` transformer (alpine → internal-registry image); wire the corrected sandbox patch. | -| `deploy/knative/overlays/ocp/patch-sandbox.yaml` | Rewrite to patch the **`Sandbox` CR** `podTemplate` (§5.2): security context, `serviceAccountName`, `command: [sleep infinity]`; **drop the `emptyDir` override** so `volumeClaimTemplates` backs `/workspace`. | -| `deploy/knative/setup-ocp.sh` | Add: agent-sandbox v0.5.0 controller install (`kubectl apply --server-side -f …/v0.5.0/manifest.yaml` + `kubectl wait --for=condition=Established crd/sandboxes.agents.x-k8s.io`); Sandbox CR apply + `.status.selector` poll → pod Ready; sandbox SA + `nonroot-v2` grant; pre-smoke check that `llm-credentials` exists. | -| `deploy/knative/README-ocp.md` | Remove `leaf-work` PVC references; document durable-PVC-via-CR + in-cluster sandbox image + the controller-install step. | + +| File | Change | +| ------------------------------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `deploy/knative/sandbox.Dockerfile` | Add `ripgrep` to the `apk add` line. | +| `deploy/knative/overlays/ocp/kustomization.yaml` | Remove the `../../leaf-pvc.yaml` resource; keep the `images:` transformer (alpine → internal-registry image); wire the corrected sandbox patch. | +| `deploy/knative/overlays/ocp/patch-sandbox.yaml` | Rewrite to patch the **`Sandbox` CR** `podTemplate` (§5.2): security context, `serviceAccountName`, `command: [sleep infinity]`; **drop the `emptyDir` override** so `volumeClaimTemplates` backs `/workspace`. | +| `deploy/knative/setup-ocp.sh` | Add: agent-sandbox v0.5.0 controller install (`kubectl apply --server-side -f …/v0.5.0/manifest.yaml` + `kubectl wait --for=condition=Established crd/sandboxes.agents.x-k8s.io`); Sandbox CR apply + `.status.selector` poll → pod Ready; sandbox SA + `nonroot-v2` grant; pre-smoke check that `llm-credentials` exists. | +| `deploy/knative/README-ocp.md` | Remove `leaf-work` PVC references; document durable-PVC-via-CR + in-cluster sandbox image + the controller-install step. | Log output for all long commands → `/tmp/sh/p0prime/*.log` (per repo Context Budget rules). ### 5.4 Deploy sequence (`setup-ocp.sh`, post-change) + 1. OpenShift Serverless operator + `KnativeServing` CR (PVC/securitycontext feature flags) — **exists**. 2. **NEW:** agent-sandbox v0.5.0 controller + CRD (`kubectl wait … Established`). 3. Optional KEDA (`--with-keda`) — exists; **not required** for this slice (sync `/runs` smoke). @@ -161,7 +169,7 @@ Log output for all long commands → `/tmp/sh/p0prime/*.log` (per repo Context B Executed in two stages to de-risk the largest unknown first: -**Stage 1 — walking skeleton (sandbox tier alone).** Install the controller and apply *only* the +**Stage 1 — walking skeleton (sandbox tier alone).** Install the controller and apply _only_ the Sandbox CR. Confirm: the PVC binds (`gp3-csi`, `WaitForFirstConsumer` → binds on pod schedule); the pod runs **non-root** as 65532; `/workspace` is writable (`kubectl exec sandbox-0 -- sh -c 'touch /workspace/.probe'`). This proves the v0.5.0 controller propagates `volumeClaimTemplates` + the @@ -178,15 +186,17 @@ All long smoke/kubectl output → `/tmp/sh/p0prime/*.log`, analyzed via subagent logs into the main context). ## 7. Risks & mitigations -| Risk | Mitigation | -|------|-----------| + +| Risk | Mitigation | +| ---------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | agent-sandbox v0.5.0 controller may not propagate `podTemplate` `fsGroup`/`runAsUser`/`serviceAccountName` from the CR | **Stage-1 walking skeleton catches it first**, before any harness work. If not propagated, patch the generated pod directly or fall back to a Sandbox-CR field the controller does honor. | -| Kustomize can't cleanly merge a container inside a CRD `podTemplate` | Use a **JSON6902** patch with explicit paths (§5.2); verify with `kustomize build` render before applying. | -| EBS `WaitForFirstConsumer` → PVC `Pending` until pod schedules | The readiness poll tolerates the `Pending` → `Bound` window (matches Kind's selector-then-Ready poll shape). | -| GHCR harness image stale (pre-P1) | `build.yaml` publishes `serverless-harness` on **every push to `main`**; P1 is merged, so `:latest` is post-P1. Confirm image digest/date in Stage 2 if the smoke misbehaves. | -| `restricted-v2` didn't cleanly inject a UID for the harness (ocp-setup memory) | We **pin** `runAsUser: 65532` and bind under **`nonroot-v2`** (not `restricted-v2`), sidestepping SCC UID injection entirely. | +| Kustomize can't cleanly merge a container inside a CRD `podTemplate` | Use a **JSON6902** patch with explicit paths (§5.2); verify with `kustomize build` render before applying. | +| EBS `WaitForFirstConsumer` → PVC `Pending` until pod schedules | The readiness poll tolerates the `Pending` → `Bound` window (matches Kind's selector-then-Ready poll shape). | +| GHCR harness image stale (pre-P1) | `build.yaml` publishes `serverless-harness` on **every push to `main`**; P1 is merged, so `:latest` is post-P1. Confirm image digest/date in Stage 2 if the smoke misbehaves. | +| `restricted-v2` didn't cleanly inject a UID for the harness (ocp-setup memory) | We **pin** `runAsUser: 65532` and bind under **`nonroot-v2`** (not `restricted-v2`), sidestepping SCC UID injection entirely. | ## 8. Out of scope / explicit non-goals + - Shared sandbox pool, N:M routing, RWX on the sandbox tier (P2, #46). - Kata isolation, ratio experiments (P3, #48). - `leaf-orchestrator.yaml`, gate/cron smokes (separate P1 follow-up). @@ -195,4 +205,4 @@ logs into the main context). --- -*Assisted-By: Claude (Anthropic AI) * +_Assisted-By: Claude (Anthropic AI) _ diff --git a/docs/specs/2026-07-02-p1-fs-free-harness-design.md b/docs/specs/2026-07-02-p1-fs-free-harness-design.md index 73de6f9..5496512 100644 --- a/docs/specs/2026-07-02-p1-fs-free-harness-design.md +++ b/docs/specs/2026-07-02-p1-fs-free-harness-design.md @@ -38,7 +38,7 @@ harness does network I/O only** (LLM, Redis, sandbox exec API). Today the harnes Because the harness and the async worker both read/write the same `/work` PVC, and (in the epic's target) many harness pods would share it across nodes, this file coupling is what forces cross-node -**RWX** on OpenShift. The OCP RWX pain is a *symptom*; the root cause is harness filesystem I/O. +**RWX** on OpenShift. The OCP RWX pain is a _symptom_; the root cause is harness filesystem I/O. P1 removes that I/O. Once the envelope and markers are inline-or-Redis and the working set is on the sandbox's own durable volume, the harness (Service **and** async worker) mounts **no shared writable @@ -57,7 +57,7 @@ a Redis key changes nothing about integrity. P1 is an FS-surface reduction, not No live external consumer exists (the in-repo callers — `cron-dispatch.ts`, the smoke drivers — are updated in the same change; the planned BugStone Archetype-A consumer is designed but unbuilt). A -transitional "accept both shapes" is not available *to the harness*: honoring the old +transitional "accept both shapes" is not available _to the harness_: honoring the old `inputsRef`/`resultRef` fields would require keeping the `/work` mount, defeating P1. So the file paths are removed outright. @@ -70,20 +70,21 @@ process, so it does not violate FS-free. ```jsonc { - "sessionId": "run-123/i1", // correlation + idempotency key (unchanged) - "item": { // NEW — was the JSON body of the inputsRef file + "sessionId": "run-123/i1", // correlation + idempotency key (unchanged) + "item": { + // NEW — was the JSON body of the inputsRef file "item_id": "i1", - "file": "src/foo.ts", // relative to workspaceRef, resolved in the sandbox + "file": "src/foo.ts", // relative to workspaceRef, resolved in the sandbox "pattern": "eval(", - "require_approval": false + "require_approval": false, }, - "decision": { "gateId": 1, "action": "approve" }, // NEW, resume/approve only — was the decisionRef file - "model": "claude-haiku-4-5", // unchanged optionals + "decision": { "gateId": 1, "action": "approve" }, // NEW, resume/approve only — was the decisionRef file + "model": "claude-haiku-4-5", // unchanged optionals "provider": "anthropic", - "workspaceRef": "/workspace/run-123/repo", // absolute path INSIDE the sandbox (unchanged) + "workspaceRef": "/workspace/run-123/repo", // absolute path INSIDE the sandbox (unchanged) "maxTurns": 20, "async": false, - "tenant": "team1" + "tenant": "team1", } ``` @@ -143,7 +144,7 @@ same record. ``` - **Write:** `SET leaf:result: EX ` — value + expiry in one call. - - **Sync path:** the HTTP layer writes the record *and* returns the union, so a sync result is also + - **Sync path:** the HTTP layer writes the record _and_ returns the union, so a sync result is also queryable via `/runs/status` for the TTL window (matches today, where the sync verdict file was also readable). - **Async path:** the worker (`leaf-job-runner.ts`) writes the record instead of @@ -152,7 +153,7 @@ same record. cleanup — records self-expire, no unbounded growth. A `paused` record carries the same TTL and is overwritten (fresh TTL) on approve/resume, so a long human-gate wait is bounded by the TTL; set it higher for long gates (documented caveat). -- **Concurrency:** single writer per `sessionId` in practice (one sync request *or* one queue +- **Concurrency:** single writer per `sessionId` in practice (one sync request _or_ one queue consumer owns a leaf at a time — the work-queue `claim` guarantees a single in-flight consumer), so last-write-wins is safe; no CAS at P1. @@ -192,12 +193,12 @@ survives restarts," explicitly as an alternative to a size-1 StatefulSet, and it apiVersion: agents.x-k8s.io/v1alpha1 kind: Sandbox metadata: - name: sandbox-0 # the name is the config handle now (see §6.2) + name: sandbox-0 # the name is the config handle now (see §6.2) spec: - volumeClaimTemplates: # durable working set (survives pod restart) + volumeClaimTemplates: # durable working set (survives pod restart) - metadata: { name: workspace } spec: - accessModes: ["ReadWriteOnce"] # RWO fine at P1 (single sandbox) + accessModes: ['ReadWriteOnce'] # RWO fine at P1 (single sandbox) resources: { requests: { storage: 1Gi } } podTemplate: spec: @@ -205,7 +206,7 @@ spec: - name: sandbox image: volumeMounts: - - { name: workspace, mountPath: /workspace } # durable, was emptyDir + - { name: workspace, mountPath: /workspace } # durable, was emptyDir ``` ### 6.2 Harness exec resolution — `@sh/k8s-sandbox` @@ -218,8 +219,8 @@ the pod from a **label selector**, authoritatively, from the CR's own status: - Config becomes **Sandbox name + namespace**: `KAGENTI_SANDBOX_NAME` (+ existing `KAGENTI_SANDBOX_NAMESPACE`). `KAGENTI_SANDBOX_POD` is still honored as a fallback so nothing breaks mid-migration; if set, it short-circuits resolution. -- Resolution: read `Sandbox/` `.status.selector` (the CRD documents it as *"the label selector - for pods"*), then `kubectl get pod -l -o jsonpath='{.items[0].metadata.name}'` → pod +- Resolution: read `Sandbox/` `.status.selector` (the CRD documents it as _"the label selector + for pods"_), then `kubectl get pod -l -o jsonpath='{.items[0].metadata.name}'` → pod name → the existing `kubectl exec` path is otherwise unchanged. - **When** it resolves: once **per leaf-session**, at extension init. The k8s-sandbox extension factory is synchronous, so the pod name is resolved in the async `runLeaf`/`runTurn` setup and the @@ -284,12 +285,12 @@ the pod from a **label selector**, authoritatively, from the CR's own status: ## 10. Acceptance mapping (issue #45) -| Acceptance | Met by | -|---|---| -| Harness/worker mount no shared writable volume; red-team grep finds no `/work` mount and no `writeFileSync` in the envelope path | §7 deploy edits + §9 red-team grep | -| `leaf-smoke.sh` passes with inputs inline + verdict from response/Redis; Claim 0 still holds | §3, §8 driver rewrite | -| Async path drains via Redis; verdict retrievable via `/runs/status` | §4 worker writes `leaf:result:*`; §3.4 status reads it | -| Sandbox repo survives a sandbox pod restart (durable PVC) | §6.1 `volumeClaimTemplates` + §9 durability smoke | +| Acceptance | Met by | +| -------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------ | +| Harness/worker mount no shared writable volume; red-team grep finds no `/work` mount and no `writeFileSync` in the envelope path | §7 deploy edits + §9 red-team grep | +| `leaf-smoke.sh` passes with inputs inline + verdict from response/Redis; Claim 0 still holds | §3, §8 driver rewrite | +| Async path drains via Redis; verdict retrievable via `/runs/status` | §4 worker writes `leaf:result:*`; §3.4 status reads it | +| Sandbox repo survives a sandbox pod restart (durable PVC) | §6.1 `volumeClaimTemplates` + §9 durability smoke | ## 11. Non-goals (this phase) @@ -311,4 +312,4 @@ deliverable (P0′ — manifests here are written OCP-aware, but the cutover is --- -*Assisted-By: Claude (Anthropic AI) * +_Assisted-By: Claude (Anthropic AI) _ diff --git a/docs/specs/2026-07-02-p2-shared-sandbox-pool-design.md b/docs/specs/2026-07-02-p2-shared-sandbox-pool-design.md index e319ac3..e54ba26 100644 --- a/docs/specs/2026-07-02-p2-shared-sandbox-pool-design.md +++ b/docs/specs/2026-07-02-p2-shared-sandbox-pool-design.md @@ -19,7 +19,7 @@ That is 1:1. P2 makes it **N:M**: many short-lived, fresh-context leaf harnesses load-balance across a **static pool of N sandbox pods**, so the dense, cheap harness tier is decoupled from the smaller, durable sandbox tier. The exact -sharing ratio (~20:1) is **not** fixed here — it is an empirical P3 experiment. P2 delivers the *mechanism*: pod +sharing ratio (~20:1) is **not** fixed here — it is an empirical P3 experiment. P2 delivers the _mechanism_: pod discovery, lease-based assignment, per-sandbox repo distribution, and per-leaf worktree isolation. `kubectl exec` targets a **concrete pod name**, so "pick the first Running pod" must become deliberate, @@ -29,13 +29,13 @@ load-aware pod selection. That is the heart of P2. These were settled during brainstorming and are not relitigated here: -| # | Decision | Rationale | -|---|----------|-----------| -| D1 | **Storage topology: per-sandbox RWO copy.** Each sandbox pod holds its own repo copy on its own RWO PVC. **No RWX** on the deployable path. RWX (fleet-wide single repo) is documented as the alternative only. | Runs on the EBS-only OCP 4.20 cluster today; matches "the harness mounts nothing"; RWX is heavier infra (EFS/CSI), slower networked FS, cross-pod contention. | -| D2 | **Routing: harness-side pick + Redis leases.** Selection logic stays in the harness/`@sh/k8s-sandbox` layer; Redis holds per-pod lease counters for least-loaded + capacity backpressure. **No new deployable component.** | Reuses the existing hard Redis dependency; crash-safe lease reclaim via TTL mirrors the existing leaf-resume model. | -| D3 | **Repo seeding: ref-pinned lazy converge.** The envelope carries a git ref; on leaf start the leased pod's repo is fetched/converged to that ref, then a worktree is created. Idempotent (pod already at ref = no-op). **Eager pre-warm deferred to P3.** | Guarantees batch-wide commit consistency regardless of which pod a leaf lands on; amortizes clone cost across the sharing ratio; survives pod churn for free. | -| D4 | **Pool scaling: static N, config knob.** N `Sandbox` CRs declared in kustomize; N and per-pod cap are values tuned empirically in P3. **Autoscaling is future work only.** | Deterministic, no new controller; saturation backpressures through the existing async Redis/KEDA queue and a bounded sync wait. | -| D5 | **Soft capacity cap.** ~20 is an empirical figure, not a safety bound; rare concurrent overshoot is acceptable. | Avoids a hard-CAS hot path; the true safety boundary is Kata at the pod level (P3). | +| # | Decision | Rationale | +| --- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| D1 | **Storage topology: per-sandbox RWO copy.** Each sandbox pod holds its own repo copy on its own RWO PVC. **No RWX** on the deployable path. RWX (fleet-wide single repo) is documented as the alternative only. | Runs on the EBS-only OCP 4.20 cluster today; matches "the harness mounts nothing"; RWX is heavier infra (EFS/CSI), slower networked FS, cross-pod contention. | +| D2 | **Routing: harness-side pick + Redis leases.** Selection logic stays in the harness/`@sh/k8s-sandbox` layer; Redis holds per-pod lease counters for least-loaded + capacity backpressure. **No new deployable component.** | Reuses the existing hard Redis dependency; crash-safe lease reclaim via TTL mirrors the existing leaf-resume model. | +| D3 | **Repo seeding: ref-pinned lazy converge.** The envelope carries a git ref; on leaf start the leased pod's repo is fetched/converged to that ref, then a worktree is created. Idempotent (pod already at ref = no-op). **Eager pre-warm deferred to P3.** | Guarantees batch-wide commit consistency regardless of which pod a leaf lands on; amortizes clone cost across the sharing ratio; survives pod churn for free. | +| D4 | **Pool scaling: static N, config knob.** N `Sandbox` CRs declared in kustomize; N and per-pod cap are values tuned empirically in P3. **Autoscaling is future work only.** | Deterministic, no new controller; saturation backpressures through the existing async Redis/KEDA queue and a bounded sync wait. | +| D5 | **Soft capacity cap.** ~20 is an empirical figure, not a safety bound; rare concurrent overshoot is acceptable. | Avoids a hard-CAS hot path; the true safety boundary is Kata at the pod level (P3). | **Threat model (locked — from epic #49, not relitigated):** there is **no agent in the sandbox** — it is a passive `kubectl exec` target. The threat is a **compromised/injected harness** and **kernel exploits**. Blast @@ -48,7 +48,7 @@ agent-sandbox `v1beta1` **removed `spec.replicas`** (the `Sandbox` is a single-i an open upstream feature request, [kubernetes-sigs/agent-sandbox#34](https://github.com/kubernetes-sigs/agent-sandbox/issues/34)). So the P2 pool primitive is **N distinct `Sandbox` CRs** `sandbox-0 … sandbox-{N-1}`: -- **Own RWO PVC each** via the existing `volumeClaimTemplates` — this *is* the per-sandbox repo copy (D1). No +- **Own RWO PVC each** via the existing `volumeClaimTemplates` — this _is_ the per-sandbox repo copy (D1). No shared volume, no RWX. - **Distinct names** deliberately sidestep the agent-sandbox v0.5.0 **bare-pod name-collision / adopt-error** gotcha observed in P0′. @@ -101,7 +101,7 @@ Bounded wait with backoff, up to a timeout: ## 5. Repo lifecycle & workspace layout -The unit shared across leaves on a pod is the **git object store**, *not* a working tree — so two leaves at +The unit shared across leaves on a pod is the **git object store**, _not_ a working tree — so two leaves at different commits never contend over a checkout. - **`/workspace/repo`** — the canonical clone; its object store accumulates every ref ever fetched on that pod. @@ -113,7 +113,7 @@ different commits never contend over a checkout. - **Cleanup:** `git worktree remove` on leaf end; orphaned worktrees from crashed leaves are reclaimed by `git worktree prune` plus a dir-age sweep performed opportunistically on acquire. -This yields **batch-wide consistency** (each leaf pins its own commit regardless of pod) *and* lets a single pod +This yields **batch-wide consistency** (each leaf pins its own commit regardless of pod) _and_ lets a single pod serve many refs concurrently. ## 6. FS-free contract & leaf-flow interaction @@ -143,10 +143,10 @@ and the existing Redis-backed resume re-runs the leaf, which **re-acquires** (li Additive to the P1 wire contract: -| Field | Type | Meaning | -|-------|------|---------| -| `repoUrl` | string | Git remote to converge from. | -| `ref` | string | Commit SHA (preferred) or branch/tag; the pod converges to it and the worktree pins the resolved commit. | +| Field | Type | Meaning | +| --------- | ------ | -------------------------------------------------------------------------------------------------------- | +| `repoUrl` | string | Git remote to converge from. | +| `ref` | string | Commit SHA (preferred) or branch/tag; the pod converges to it and the worktree pins the resolved commit. | `workspaceRef` becomes **derived** (`/workspace/leaves/`) rather than a caller-supplied absolute pod path. Callers we own (leaf-orchestrator, dispatch scripts) are updated in the same cut to send `repoUrl`/`ref` and to @@ -173,13 +173,13 @@ one trust domain in P2. Documented here as the known limitation P3 resolves. ## 10. Failure modes -| Event | Behavior | -|-------|----------| -| Pod crashes mid-leaf | Lease TTL-expires → reclaimed on next acquire; worktree orphaned then age-pruned; leaf resumes (existing Redis resume) and re-acquires + re-converges (likely a different pod). | -| Pod removed from pool | K8s list stops returning it → never picked. In-flight leaf on it fails through the existing verdict-error path and resumes elsewhere. | -| All pods FULL | §4.3 — async stays queued; sync gets `503 Retry-After`. | -| `git fetch`/converge fails | Leaf errors out via the existing verdict-error path; lease released; resume retries. | -| Two harnesses race one pod | Soft-cap overshoot by a small margin (D5) — accepted. | +| Event | Behavior | +| -------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| Pod crashes mid-leaf | Lease TTL-expires → reclaimed on next acquire; worktree orphaned then age-pruned; leaf resumes (existing Redis resume) and re-acquires + re-converges (likely a different pod). | +| Pod removed from pool | K8s list stops returning it → never picked. In-flight leaf on it fails through the existing verdict-error path and resumes elsewhere. | +| All pods FULL | §4.3 — async stays queued; sync gets `503 Retry-After`. | +| `git fetch`/converge fails | Leaf errors out via the existing verdict-error path; lease released; resume retries. | +| Two harnesses race one pod | Soft-cap overshoot by a small margin (D5) — accepted. | ## 11. Testing @@ -193,16 +193,16 @@ one trust domain in P2. Documented here as the known limitation P3 resolves. ## 12. Acceptance mapping (issue #46) -| Issue #46 scope item | Addressed by | -|----------------------|--------------| -| Replace fixed `KAGENTI_SANDBOX_POD` with pool assignment (pod-selection logic) | §4 routing + §8 `KAGENTI_SANDBOX_POOL_SELECTOR` | -| Within a sandbox: shared repo (read-mostly), per-leaf isolated worktrees, `/workspace//…` layout | §5 shared object store + per-leaf worktree | -| Across sandboxes: storage topology — RWX vs per-sandbox copy (design both) | D1 + §3 (per-sandbox RWO chosen; RWX = §13 alternative) | -| Sandbox tier as a managed set with durable storage | §3 N `Sandbox` CRs, per-CR RWO PVC | -| Open Q — routing mechanism | §4 (harness-side pick + Redis leases) | -| Open Q — repo distribution | D3 + §5 (ref-pinned lazy converge) | -| Open Q — sandbox lifecycle/scaling | D4 + §8 (static N config knob) | -| Open Q — isolation between shared leaves | §9 (worktree/dir, Kata → P3) | +| Issue #46 scope item | Addressed by | +| ----------------------------------------------------------------------------------------------------- | ------------------------------------------------------- | +| Replace fixed `KAGENTI_SANDBOX_POD` with pool assignment (pod-selection logic) | §4 routing + §8 `KAGENTI_SANDBOX_POOL_SELECTOR` | +| Within a sandbox: shared repo (read-mostly), per-leaf isolated worktrees, `/workspace//…` layout | §5 shared object store + per-leaf worktree | +| Across sandboxes: storage topology — RWX vs per-sandbox copy (design both) | D1 + §3 (per-sandbox RWO chosen; RWX = §13 alternative) | +| Sandbox tier as a managed set with durable storage | §3 N `Sandbox` CRs, per-CR RWO PVC | +| Open Q — routing mechanism | §4 (harness-side pick + Redis leases) | +| Open Q — repo distribution | D3 + §5 (ref-pinned lazy converge) | +| Open Q — sandbox lifecycle/scaling | D4 + §8 (static N config knob) | +| Open Q — isolation between shared leaves | §9 (worktree/dir, Kata → P3) | ## 13. Non-goals (this phase) @@ -223,4 +223,4 @@ one trust domain in P2. Documented here as the known limitation P3 resolves. --- -*Assisted-By: Claude (Anthropic AI) — brainstorming + spec authoring for P2.* +_Assisted-By: Claude (Anthropic AI) — brainstorming + spec authoring for P2._ diff --git a/docs/specs/2026-07-03-e6-workload-parameterized-sandbox-load-design.md b/docs/specs/2026-07-03-e6-workload-parameterized-sandbox-load-design.md index 0cb25b9..dec7f64 100644 --- a/docs/specs/2026-07-03-e6-workload-parameterized-sandbox-load-design.md +++ b/docs/specs/2026-07-03-e6-workload-parameterized-sandbox-load-design.md @@ -6,7 +6,7 @@ Scope: Hardens the **P3** sandbox sharing-ratio experiment ([`2026-07-03-p3-sandbox-sharing-ratio-experiments-design.md`](2026-07-03-p3-sandbox-sharing-ratio-experiments-design.md), merged PR #58). Resolves [#62](https://github.com/kagenti/serverless-harness/issues/62) (noise-sensitive knee) **and** a deeper validity gap surfaced in review: the E6/E7 leaf workload is a trivial -`marker.txt` check, so the reported ratio N ≈ 29–48:1 is an optimistic *upper bound*, not a +`marker.txt` check, so the reported ratio N ≈ 29–48:1 is an optimistic _upper bound_, not a representative Archetype-A figure. Builds on the two-tier epic ([#49](https://github.com/kagenti/serverless-harness/issues/49)). @@ -30,7 +30,7 @@ P3 measured the harness→sandbox sharing ratio as N ≈ 1/duty, where duty = pe make a single rung dip, tripping the break early (observed on OCP: c=4 0.213 < c=2 0.245 → knee=2, floor=fail — a noise artifact, not saturation). Compounding it, the harness ksvc is `max-scale: 5` + `containerConcurrency: 1`, so **at most 5 leaves ever reach the pinned sandbox - concurrently** — any knee above 5 measures the *harness* tier's cap, not the sandbox's. + concurrently** — any knee above 5 measures the _harness_ tier's cap, not the sandbox's. This spec makes the headline output an **N-vs-workload curve** (confound-free, measured at C=1 across real Archetype-A leaves of increasing intensity) and fixes the concurrency-sweep confounds so the knee, @@ -38,13 +38,13 @@ where reported, is the sandbox's and is noise-robust. ## 2. Decisions locked (brainstorm 2026-07-03) -| # | Decision | Rationale | -|---|----------|-----------| -| D1 | **Headline = N-vs-workload curve, measured at C=1.** Report N ≈ 1/duty across a range of workload intensities, each point tagged with the per-leaf sandbox exec count/ms. | The C=1 duty measurement has no concurrency confounds (max-scale/noise are irrelevant at one leaf); the curve answers "what ratio does Archetype A get" honestly, as a range tied to leaf intensity. | -| D2 | **Real Archetype-A workload variants L0/L1/L2 by review scope.** L0 light (one small file, one finding) → L1 (larger file) → L2 heavy (multi-file / multi-pattern review). Genuine structured code-review leaves over real fixtures, not marker checks. | Review scope is the intensity axis the user chose; it maps directly to how many sandbox tool calls a leaf makes, which is what drives duty. Reuses the canonical Archetype-A code-review shape already in the repo. | -| D3 | **Fix the concurrency sweep (the original #62 knee):** raise harness `max-scale` for the sweep so the sandbox is the concurrency limiter; warm `min-scale` baseline; multi-sample rungs (median); smooth `detectKnee` to break only on a *sustained* decline. Run the sweep at the **heaviest** variant. | Removes the three knee confounds (harness cap, cold-start, single-dip). The heaviest leaf stresses the sandbox most, so a real knee (if any) is most visible there. | -| D4 | **Report N always with its workload intensity; knee as a floor with its max-scale bound.** Supersede P3's single-N claim (the light-workload upper bound). | A ratio without a stated workload is misleading; the review that prompted this made exactly that point. | -| D5 | **No harness/leaf code changes to inject synthetic work.** Intensity comes from the *item* (which file, how many patterns) and the emergent tool calls; measure the *actual* exec count per leaf. | Keeps the FS-free harness and leaf contract untouched; the timing hook already records real sandbox work, so measured (not assumed) intensity anchors each curve point. | +| # | Decision | Rationale | +| --- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| D1 | **Headline = N-vs-workload curve, measured at C=1.** Report N ≈ 1/duty across a range of workload intensities, each point tagged with the per-leaf sandbox exec count/ms. | The C=1 duty measurement has no concurrency confounds (max-scale/noise are irrelevant at one leaf); the curve answers "what ratio does Archetype A get" honestly, as a range tied to leaf intensity. | +| D2 | **Real Archetype-A workload variants L0/L1/L2 by review scope.** L0 light (one small file, one finding) → L1 (larger file) → L2 heavy (multi-file / multi-pattern review). Genuine structured code-review leaves over real fixtures, not marker checks. | Review scope is the intensity axis the user chose; it maps directly to how many sandbox tool calls a leaf makes, which is what drives duty. Reuses the canonical Archetype-A code-review shape already in the repo. | +| D3 | **Fix the concurrency sweep (the original #62 knee):** raise harness `max-scale` for the sweep so the sandbox is the concurrency limiter; warm `min-scale` baseline; multi-sample rungs (median); smooth `detectKnee` to break only on a _sustained_ decline. Run the sweep at the **heaviest** variant. | Removes the three knee confounds (harness cap, cold-start, single-dip). The heaviest leaf stresses the sandbox most, so a real knee (if any) is most visible there. | +| D4 | **Report N always with its workload intensity; knee as a floor with its max-scale bound.** Supersede P3's single-N claim (the light-workload upper bound). | A ratio without a stated workload is misleading; the review that prompted this made exactly that point. | +| D5 | **No harness/leaf code changes to inject synthetic work.** Intensity comes from the _item_ (which file, how many patterns) and the emergent tool calls; measure the _actual_ exec count per leaf. | Keeps the FS-free harness and leaf contract untouched; the timing hook already records real sandbox work, so measured (not assumed) intensity anchors each curve point. | ## 3. Workload variants (real Archetype-A code review) @@ -57,11 +57,11 @@ per-branch `marker.txt` scheme for the workload refs; E7's mixed-ref refs stay m Three ordered leaf **items** of increasing review scope, each a `LeafItem {item_id, file, pattern}` run through the unchanged review prompt: -| Variant | Item | Expected sandbox work | -|---------|------|-----------------------| -| **L0** (light) | one small file (`safe.py`), one pattern | converge + ~1 read → lightest duty, highest N | -| **L1** (medium) | a larger file, one pattern | converge + read + a grep or two | -| **L2** (heavy) | the larger file, a pattern that induces broader scanning (multi-match / context reads) | converge + several reads/greps → highest duty, lowest N | +| Variant | Item | Expected sandbox work | +| --------------- | -------------------------------------------------------------------------------------- | ------------------------------------------------------- | +| **L0** (light) | one small file (`safe.py`), one pattern | converge + ~1 read → lightest duty, highest N | +| **L1** (medium) | a larger file, one pattern | converge + read + a grep or two | +| **L2** (heavy) | the larger file, a pattern that induces broader scanning (multi-match / context reads) | converge + several reads/greps → highest duty, lowest N | Intensity is **emergent** (model-driven tool use), so it is **measured, not assumed** (§4): each point on the curve is tagged with the leaf's actual exec count. The variants only need to reliably span @@ -87,7 +87,7 @@ A fresh pod per measurement (drain first, per P3's C=1 fix) keeps the log scoped `WorkloadPoint = { label: string; execMs: number; execCount: number; wallMs: number }` and each output carries `{ label, execCount, duty, n }`. Pure, unit-tested (monotonicity sanity: heavier execMs at equal wall → higher duty → lower N). -- **Change `detectKnee` to break on a *sustained* decline.** New signature +- **Change `detectKnee` to break on a _sustained_ decline.** New signature `detectKnee(points, degradeX, patience = 2)`: track the running-max throughput; a rung is "healthy" if `p95 ≤ degradeX·baseline` **and** its throughput ≥ the running max (i.e. still at or above the best seen). Advance the knee on healthy rungs; only **break after `patience` consecutive unhealthy rungs**, @@ -144,4 +144,4 @@ supersedes the prior single-N claim, framing it explicitly as the L0 (near-empty --- -*Assisted-By: Claude (Anthropic AI) — brainstorming + spec authoring for the E6 workload-parameterization hardening.* +_Assisted-By: Claude (Anthropic AI) — brainstorming + spec authoring for the E6 workload-parameterization hardening._ diff --git a/docs/specs/2026-07-03-p3-sandbox-sharing-ratio-experiments-design.md b/docs/specs/2026-07-03-p3-sandbox-sharing-ratio-experiments-design.md index b85cba5..2b1e551 100644 --- a/docs/specs/2026-07-03-p3-sandbox-sharing-ratio-experiments-design.md +++ b/docs/specs/2026-07-03-p3-sandbox-sharing-ratio-experiments-design.md @@ -8,15 +8,15 @@ pool + routing, [#46](https://github.com/kagenti/serverless-harness/issues/46)) [#47](https://github.com/kagenti/serverless-harness/issues/47)), both merged. Measures the harness→sandbox **sharing ratio** on the runc runtime to set the pool's capacity knobs. **Kata/VM isolation and intra-pod hardening are split out to a new [P4 (#57)](https://github.com/kagenti/serverless-harness/issues/57)** and are -*not* in scope here. +_not_ in scope here. --- ## 1. Goal & motivation -P2 delivered the *mechanism* for many leaf harnesses to share a small pool of sandbox pods (N distinct `Sandbox` +P2 delivered the _mechanism_ for many leaf harnesses to share a small pool of sandbox pods (N distinct `Sandbox` CRs, harness-side Redis-lease routing, ref-pinned lazy converge, per-leaf worktrees). It deliberately left the -**capacity numbers** unmeasured: `KAGENTI_SANDBOX_CAP` defaults to a *soft* 20 and pool size N is a static config +**capacity numbers** unmeasured: `KAGENTI_SANDBOX_CAP` defaults to a _soft_ 20 and pool size N is a static config knob, both tagged "tune empirically in P3." P3 produces those numbers. It answers, with evidence on a representative cluster: @@ -39,7 +39,7 @@ The 2026-07-03 brainstorm reversed that dependency and split the phase: - **The ratio baseline does not need Kata.** Concurrency knee, converge contention, CPU/mem per leaf, and CAP/N tuning are all properties of the workload on whatever runtime the pods use. They are measured on **runc** (what - we have). Kata only adds a **startup/overhead delta** that P4 measures *on top of* this baseline — so the + we have). Kata only adds a **startup/overhead delta** that P4 measures _on top of_ this baseline — so the baseline is a prerequisite for the Kata work, not the other way round. - **Kata cannot run on the live cluster as-is.** The OCP 4.20.8 cluster is all `m6i.xlarge` — standard EC2 with **no `/dev/kvm`** (nested virtualization is not exposed on non-`.metal` instances), so default Kata (QEMU/KVM) @@ -53,23 +53,23 @@ moves to **P4 (#57)**, gated on the infra spike. §8 hands P4 a clean starting p These were settled during brainstorming and are not relitigated in planning: -| # | Decision | Rationale | -|---|----------|-----------| -| D1 | **Split P3 (experiments) from P4 (Kata/isolation).** P3 measures the ratio on runc; P4 owns VM isolation + intra-pod hardening. | Ratio baseline is runtime-independent and unblocked; Kata is infra-blocked on this cluster. Decouples shippable work from a spike. | -| D2 | **Ratio meaning: concurrency cap is primary; N is derived.** Measure the per-sandbox concurrency knee (→ `CAP`); compute the provisioning ratio N from the observed per-leaf duty cycle in the **same** run. | One load experiment yields both knobs; avoids a second time-averaged fleet workload. | -| D3 | **Experiment set: E6 (saturation curve) + E7 (converge contention + mixed-ref correctness) + a light feed-back check.** No standalone fleet/CAP experiment. | E6+E7 yield CAP, N, the bottleneck, and the deferred mixed-ref validation. P2 already proved pool spread / never-over-cap / self-heal, so the feed-back check re-runs that at the derived CAP rather than re-deriving it. | -| D4 | **Load substrate: in-cluster git-daemon pod** serving a seeded bare repo with multiple refs over `git://`. | Reachable by all pool pods, hermetic (no GitHub egress/auth), repeatable, and the only option that supports mixed-ref converge *across* the pool. `file://` is local to one pod's RWO PVC and cannot be shared. | -| D5 | **Cluster strategy: develop on Kind, authoritative numbers on OCP.** Iterate drivers on the standing Kind `sh-knative` 3-pod pool; take the reported ratio/CAP on live OCP 4.20. | Kind is fast/deterministic but laptop-bound (absolute leaves/sec unrepresentative); OCP gives representative CPU/mem + real EBS RWO + API-server exec. Mirrors the P2 Kind-integration → gated-OCP-live pattern. | -| D6 | **Gates: hard correctness gates + reported ratio with a sanity floor.** Correctness (mixed-ref consistency, never-over-cap) fails the build; the ratio/CAP is a reported deliverable guarded by a documented minimum-concurrency floor. | A measurement experiment has no honest binary threshold; the floor still catches a silent capacity collapse in CI. | +| # | Decision | Rationale | +| --- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| D1 | **Split P3 (experiments) from P4 (Kata/isolation).** P3 measures the ratio on runc; P4 owns VM isolation + intra-pod hardening. | Ratio baseline is runtime-independent and unblocked; Kata is infra-blocked on this cluster. Decouples shippable work from a spike. | +| D2 | **Ratio meaning: concurrency cap is primary; N is derived.** Measure the per-sandbox concurrency knee (→ `CAP`); compute the provisioning ratio N from the observed per-leaf duty cycle in the **same** run. | One load experiment yields both knobs; avoids a second time-averaged fleet workload. | +| D3 | **Experiment set: E6 (saturation curve) + E7 (converge contention + mixed-ref correctness) + a light feed-back check.** No standalone fleet/CAP experiment. | E6+E7 yield CAP, N, the bottleneck, and the deferred mixed-ref validation. P2 already proved pool spread / never-over-cap / self-heal, so the feed-back check re-runs that at the derived CAP rather than re-deriving it. | +| D4 | **Load substrate: in-cluster git-daemon pod** serving a seeded bare repo with multiple refs over `git://`. | Reachable by all pool pods, hermetic (no GitHub egress/auth), repeatable, and the only option that supports mixed-ref converge _across_ the pool. `file://` is local to one pod's RWO PVC and cannot be shared. | +| D5 | **Cluster strategy: develop on Kind, authoritative numbers on OCP.** Iterate drivers on the standing Kind `sh-knative` 3-pod pool; take the reported ratio/CAP on live OCP 4.20. | Kind is fast/deterministic but laptop-bound (absolute leaves/sec unrepresentative); OCP gives representative CPU/mem + real EBS RWO + API-server exec. Mirrors the P2 Kind-integration → gated-OCP-live pattern. | +| D6 | **Gates: hard correctness gates + reported ratio with a sanity floor.** Correctness (mixed-ref consistency, never-over-cap) fails the build; the ratio/CAP is a reported deliverable guarded by a documented minimum-concurrency floor. | A measurement experiment has no honest binary threshold; the floor still catches a silent capacity collapse in CI. | ## 4. What is measured (metrics & definitions) -| Metric | Definition | Feeds | -|--------|------------|-------| -| **Concurrency knee** | The concurrent-leaf count `C*` beyond which aggregate throughput (leaves/sec) stops rising and/or per-leaf p95 latency crosses a documented degradation multiple of the `C=1` baseline. | Recommended **`KAGENTI_SANDBOX_CAP`** (`≈ C*`, with a safety margin below the hard-degradation point). | -| **Per-leaf duty cycle** `d` | Sandbox-busy time (sum of exec durations attributable to a leaf: converge + worktree + tool ops) ÷ leaf wall-clock. | **Provisioning ratio** `N_harness : N_sandbox ≈ 1/d` at the knee; guides pool size N for an expected fleet. | -| **Converge wait** | Time a leaf's `git fetch` spends blocked on the per-pod `flock` vs. executing, as concurrency rises. | Whether the object-store lock (fixable) or CPU/exec (a real ceiling) caps E6. | -| **Sandbox pod CPU/mem** | `kubectl top pod` (and/or cgroup readings) for the sandbox pod at each concurrency level. | Confirms the knee's physical cause; sizing guidance for sandbox pod resources. | +| Metric | Definition | Feeds | +| --------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------- | +| **Concurrency knee** | The concurrent-leaf count `C*` beyond which aggregate throughput (leaves/sec) stops rising and/or per-leaf p95 latency crosses a documented degradation multiple of the `C=1` baseline. | Recommended **`KAGENTI_SANDBOX_CAP`** (`≈ C*`, with a safety margin below the hard-degradation point). | +| **Per-leaf duty cycle** `d` | Sandbox-busy time (sum of exec durations attributable to a leaf: converge + worktree + tool ops) ÷ leaf wall-clock. | **Provisioning ratio** `N_harness : N_sandbox ≈ 1/d` at the knee; guides pool size N for an expected fleet. | +| **Converge wait** | Time a leaf's `git fetch` spends blocked on the per-pod `flock` vs. executing, as concurrency rises. | Whether the object-store lock (fixable) or CPU/exec (a real ceiling) caps E6. | +| **Sandbox pod CPU/mem** | `kubectl top pod` (and/or cgroup readings) for the sandbox pod at each concurrency level. | Confirms the knee's physical cause; sizing guidance for sandbox pod resources. | **The "~20:1" hypothesis** is an output to confirm or refine, never an input. The deliverable is the measured curve + a recommended CAP + a derived N, recorded with the raw numbers. @@ -164,14 +164,14 @@ P3 records the isolation starting point so P4 (#57) opens cleanly: ## 9. Failure modes & risks -| Event | Behavior / mitigation | -|-------|-----------------------| -| Knee is above the largest ladder step | Report "no knee observed below `C_max`"; extend `E6_LADDER` and re-run. CAP recommendation is then a floor, not the true ceiling — noted explicitly in RESULTS. | -| Kind resource limits dominate the curve | Expected; that is why the *authoritative* run is on OCP (D5). Kind results are reported as shape/relative only. | -| git-daemon becomes the bottleneck instead of the sandbox | Detected by E7 (converge wait would rise with git-daemon load, not pod concurrency); seed refs locally and keep the daemon read-only/cheap; note if observed. | -| Duty-cycle attribution is noisy | Report N as a range with the measured duty-cycle spread rather than a single figure; the CAP (primary) does not depend on it. | +| Event | Behavior / mitigation | +| ------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| Knee is above the largest ladder step | Report "no knee observed below `C_max`"; extend `E6_LADDER` and re-run. CAP recommendation is then a floor, not the true ceiling — noted explicitly in RESULTS. | +| Kind resource limits dominate the curve | Expected; that is why the _authoritative_ run is on OCP (D5). Kind results are reported as shape/relative only. | +| git-daemon becomes the bottleneck instead of the sandbox | Detected by E7 (converge wait would rise with git-daemon load, not pod concurrency); seed refs locally and keep the daemon read-only/cheap; note if observed. | +| Duty-cycle attribution is noisy | Report N as a range with the measured duty-cycle spread rather than a single figure; the CAP (primary) does not depend on it. | | Model/network latency inflates leaf wall-clock and deflates apparent duty cycle | Use the small fixed Archetype-A leaf and `claude-haiku-4-5`; report duty cycle from exec-time accounting, not just wall-clock, so provider latency does not distort N. | -| `llm-credentials` drift on OCP | Re-provision api-key-only before the authoritative run (P0′ lesson). | +| `llm-credentials` drift on OCP | Re-provision api-key-only before the authoritative run (P0′ lesson). | ## 10. Testing @@ -212,4 +212,4 @@ P3 records the isolation starting point so P4 (#57) opens cleanly: --- -*Assisted-By: Claude (Anthropic AI) — brainstorming + spec authoring for P3.* +_Assisted-By: Claude (Anthropic AI) — brainstorming + spec authoring for P3._ diff --git a/docs/specs/2026-07-08-sandbox-transport-grpc-design.md b/docs/specs/2026-07-08-sandbox-transport-grpc-design.md index 4b41894..f59eb27 100644 --- a/docs/specs/2026-07-08-sandbox-transport-grpc-design.md +++ b/docs/specs/2026-07-08-sandbox-transport-grpc-design.md @@ -17,7 +17,7 @@ the connection into the pod through the kube API**. This only works when the har reach the sandbox's API server, which rules out sandboxes behind NAT, on-prem, on a laptop, or in another cloud. -**Goal:** let a sandbox live anywhere by inverting connectivity — the sandbox dials *out* +**Goal:** let a sandbox live anywhere by inverting connectivity — the sandbox dials _out_ to a broker the harness also talks to — with a protocol that is **language-neutral** (any runtime can host a worker) and **firewall-friendly** (a single outbound TLS connection on `:443`), without changing the Pi orchestration loop, the session backend, @@ -26,7 +26,7 @@ or the leaf queue. ### Drivers (priority order) 1. **Bring-your-own (untrusted 3rd-party) sandbox** — external parties host their own - sandbox and register it. *Top priority.* + sandbox and register it. _Top priority._ 2. **Decoupling / heterogeneity** — a clean, language-independent harness↔sandbox contract so a sandbox can be any runtime (VM, Firecracker, remote Docker) in any language. @@ -36,7 +36,7 @@ or the leaf queue. ## 2. Key decisions (settled during brainstorming) - **Brain stays central.** The Pi loop + LLM calls remain in the harness; only command - execution is delegated. This is the *trust-correct* choice for driver #1: the LLM key + execution is delegated. This is the _trust-correct_ choice for driver #1: the LLM key and control loop never leave the harness; an untrusted sandbox only ever receives commands and returns bytes. - **The contract is a Protobuf IDL, not a TypeScript interface.** Any language with gRPC @@ -45,8 +45,8 @@ or the leaf queue. real. - **gRPC-native over HTTP/2 on `:443`.** One outbound TLS connection carrying a full-duplex bidirectional stream. HTTP/2-on-443 traverses most modern egress and NAT. - *(Connect / HTTP-1.1 fallback was considered and rejected — see §5 — because our - streaming core is full-duplex, which requires HTTP/2 regardless.)* + _(Connect / HTTP-1.1 fallback was considered and rejected — see §5 — because our + streaming core is full-duplex, which requires HTTP/2 regardless.)_ - **A single-replica relay, presence-only.** A new in-cluster process bridges the worker's outbound stream to the harness's in-cluster calls. It does **not** own matching or leasing — it mirrors connected workers into the **existing sandbox pool**, @@ -55,7 +55,7 @@ or the leaf queue. droppable into any sandbox image — and the honest proof that the contract is genuinely language-neutral rather than secretly TS-shaped. - **Per-sandbox bearer token at the edge.** The worker authenticates on connect; - SPIFFE/mTLS for untrusted BYO upgrades into the *same* seam later. + SPIFFE/mTLS for untrusted BYO upgrades into the _same_ seam later. - **Latency posture: mixed.** `kubectl-exec` stays as the fast in-cluster implementation; the remote path is added behind the same interface and degrades gracefully. @@ -86,25 +86,28 @@ Sandbox (laptop / on-prem / other cloud / same cluster) ### Component boundaries -| Unit | Purpose | Depends on | -|------|---------|-----------| -| `SandboxTransport` (interface) | The exec seam Pi sees. No transport knowledge above it. | — | -| `KubectlTransport` | Local/in-cluster fast path. Rename of today's `kubectlExecInPod`. | kubectl | -| `GrpcRelayTransport` | Harness side: turn one `exec()` into a `SandboxExec.Exec` call + stream reassembly + correlation. | relay (in-cluster gRPC) | -| `relay` | Bridge worker's outbound `Attach` stream ↔ harness `Exec`; mirror presence into the pool. | gRPC, Redis pool | -| Go worker (reference) | Sandbox side: dial `Attach`, run commands locally, stream frames, honor abort. | local shell | -| `select-sandbox` (existing) | Now returns a **transport**, not a pod name. Sees pods + remote records; leases least-loaded. | pool/lease logic | +| Unit | Purpose | Depends on | +| ------------------------------ | ------------------------------------------------------------------------------------------------- | ----------------------- | +| `SandboxTransport` (interface) | The exec seam Pi sees. No transport knowledge above it. | — | +| `KubectlTransport` | Local/in-cluster fast path. Rename of today's `kubectlExecInPod`. | kubectl | +| `GrpcRelayTransport` | Harness side: turn one `exec()` into a `SandboxExec.Exec` call + stream reassembly + correlation. | relay (in-cluster gRPC) | +| `relay` | Bridge worker's outbound `Attach` stream ↔ harness `Exec`; mirror presence into the pool. | gRPC, Redis pool | +| Go worker (reference) | Sandbox side: dial `Attach`, run commands locally, stream frames, honor abort. | local shell | +| `select-sandbox` (existing) | Now returns a **transport**, not a pod name. Sees pods + remote records; leases least-loaded. | pool/lease logic | ### The harness-facing interface (unchanged from PR #78) ```ts interface SandboxTransport { - exec(command: string, opts?: { - stdin?: Buffer; - onData?: (chunk: Buffer) => void; - signal?: AbortSignal; - timeout?: number; // seconds - }): Promise<{ stdout: Buffer; exitCode: number | null }>; + exec( + command: string, + opts?: { + stdin?: Buffer; + onData?: (chunk: Buffer) => void; + signal?: AbortSignal; + timeout?: number; // seconds + }, + ): Promise<{ stdout: Buffer; exitCode: number | null }>; close(): Promise; } ``` @@ -129,7 +132,7 @@ implementations of this one interface. The Pi orchestration loop, `run-turn`, the session backend (`RedisSessionBackend`), the leaf queue (`@sh/work-queue`), and the sandbox **pool/lease** logic. The change slots -strictly *below* the current `ExecInPod` call sites (`converge.ts`, `run-leaf.ts`, +strictly _below_ the current `ExecInPod` call sites (`converge.ts`, `run-leaf.ts`, `run-turn.ts`, `select-sandbox.ts`). ## 4. The Protobuf contract (`sandbox/v1`) @@ -217,7 +220,7 @@ message AbortResponse {} The constraint: the **worker can only dial out**, but the **harness must push commands to the worker**. A single **bidirectional streaming RPC where the worker is the client** -resolves this — commands flow to the worker on the *server→client* half of the stream the +resolves this — commands flow to the worker on the _server→client_ half of the stream the worker itself opened: ``` @@ -233,7 +236,7 @@ outbound-only property we want, expressed in a standard RPC primitive. **Why gRPC-native and not Connect.** Connect's headline advantage is a plain-HTTP/1.1 fallback that traverses HTTP-inspecting proxies. But **full-duplex bidi streaming requires HTTP/2 regardless of framework** — Connect only offers unary + server-streaming -over HTTP/1.1. Since our core *is* a full-duplex stream, Connect buys us little for the +over HTTP/1.1. Since our core _is_ a full-duplex stream, Connect buys us little for the part that matters while adding a second toolchain. We therefore use gRPC-native over HTTP/2 on `:443`. If a concrete "must traverse HTTP/1.1-only proxy" requirement ever appears, the fallback is to decompose `Attach` into a server-streaming "receive commands" @@ -249,7 +252,7 @@ A new in-cluster `Deployment`, **one replica**. It is a matchmaker-free byte bri 2. **Mirrors presence into the existing Redis sandbox pool.** It writes a lightweight record (`sandbox_id`, labels, capabilities, `capacity_max`, `transport:"grpc"`) into the same pool store `select-sandbox` already reads, and **removes it when the stream - closes**. The live `Attach` stream *is* the registration — no separate heartbeat key, + closes**. The live `Attach` stream _is_ the registration — no separate heartbeat key, no reaper. 3. **Routes `SandboxExec.Exec`.** Looks up the live stream by `sandbox_id`, sends `ServerFrame{Exec}`, and forwards the worker's `Chunk`/`End`/`Error` back as the @@ -306,7 +309,7 @@ one-in-flight execs. ## 8. Wire semantics (correlation, dedup, timeout, output cap, abort) -The frame *semantics* are carried from the superseded design verbatim — only the encoding +The frame _semantics_ are carried from the superseded design verbatim — only the encoding (protobuf) and transport (gRPC bidi) changed. - **Correlation & ordering.** Each `exec()` gets a `req_id` unique across harness @@ -320,7 +323,7 @@ The frame *semantics* are carried from the superseded design verbatim — only t the idempotency key. - **Streaming vs one-shot.** Streaming ops (bash/grep) emit `Chunk`* then `End`; the harness replays each `Chunk` into `opts.onData`, matching today's `ExecInPod` contract - byte-for-byte. Non-streaming ops (read/write) differ only in *when* bytes leave: the + byte-for-byte. Non-streaming ops (read/write) differ only in _when_ bytes leave: the worker withholds them until exit and then emits `ChunkSize`-capped `Chunk` frames followed by `End`. `streaming: false` means "no incremental delivery", not "exactly one frame" — `End` carries no payload. @@ -330,7 +333,7 @@ The frame *semantics* are carried from the superseded design verbatim — only t `cat file` yields exit 0 and empty stdout, because `End` has nowhere to put the bytes and the `Chunk`s were consumed by the original delivery. This is a known asymmetry with `KubectlTransport`, which keeps no cache and so re-runs and returns output. Callers must - not treat a dedup hit as a content read. *Honest limitation:* if the worker died mid-write, + not treat a dedup hit as a content read. _Honest limitation:_ if the worker died mid-write, exactly-once is impossible — the contract is at-least-once + dedup-by-`req_id`, and partial filesystem effects on crash are possible (same risk class as a leaf re-run today). - **Dual-ended timeout.** The worker kills its local child at `timeout_s`; the harness has @@ -348,17 +351,18 @@ The frame *semantics* are carried from the superseded design verbatim — only t **The stop mechanism differs by transport, and each declares which one it uses:** - | Transport | Mechanism | What it guarantees | - |---|---|---| - | `GrpcRelayTransport` | `remote-abort` — `Abort` for the exec's `req_id` | the worker kills the process group | - | `KubectlTransport` | `local-kill` — SIGKILL its `kubectl exec` client | the in-pod process stops on EPIPE, if at all | + | Transport | Mechanism | What it guarantees | + | --------------------- | --------------------------------------------------------------- | ---------------------------------------------- | + | `GrpcRelayTransport` | `remote-abort` — `Abort` for the exec's `req_id` | the worker kills the process group | + | `KubectlTransport` | `local-kill` — SIGKILL its `kubectl exec` client | the in-pod process stops on EPIPE, if at all | | `persistentExecInPod` | `producer-side-cap` — pod-side `head -c` in the framed pipeline | raw output cannot exceed the cap at the source | - The battery asserts the *declared* mechanism rather than accepting any truthy "stopped" + The battery asserts the _declared_ mechanism rather than accepting any truthy "stopped" signal, so a transport cannot claim one and perform another, and a deleted stop fails loudly. Neither kubectl mechanism defends against a producer that traps or ignores SIGPIPE and keeps burning CPU after we stop reading; that residual threat belongs to VM-level isolation (#57), not to the cap. + - **Abort/end races.** A late `End` for an aborted `req_id` is dropped; an `Abort` for an already-ended `req_id` is a no-op. @@ -369,7 +373,7 @@ production-behaviour decision outside this epic. - **No default deadline on the kubectl path.** `KubectlTransport` arms a timer only when `opts.timeout > 0`; `GrpcRelayTransport` always applies `DEFAULT_DEADLINE_MS` (120 s). - Pi's bash tool documents no default timeout, so *the same* model-issued `bash` with no + Pi's bash tool documents no default timeout, so _the same_ model-issued `bash` with no timeout runs unbounded on the pod path and dies at 120 s on the remote path — an unbounded pod-side exec on one backend, a surprise 120 s failure on the other. The conformance battery cannot see this: its timeout case always passes an explicit @@ -383,16 +387,16 @@ relay validates the token ↔ `sandbox_id` binding on `Attach` before parking th The worker-facing `Attach` endpoint is the **single public attack surface**; the harness-facing `SandboxExec` service is **in-cluster only**, guarded by NetworkPolicy. -| Property | Value | -|----------|-------| -| Inbound rules on sandbox | none (outbound only) | -| Egress required | `:443` only | -| Encryption | TLS 1.3 | +| Property | Value | +| ------------------------- | ------------------------------------------------ | +| Inbound rules on sandbox | none (outbound only) | +| Egress required | `:443` only | +| Encryption | TLS 1.3 | | Worker identity (day-one) | per-sandbox bearer token, scoped to `sandbox_id` | -| Public attack surface | relay `Attach` on `:443` | +| Public attack surface | relay `Attach` on `:443` | **Upgrade path for untrusted BYO.** SPIFFE/SPIRE **mTLS** with per-connection identity -slots into the *same* `Attach` endpoint — same protocol, stronger credential on the TLS +slots into the _same_ `Attach` endpoint — same protocol, stronger credential on the TLS handshake, no wire change. Deferred (§13); the bearer token is the seam it replaces. **Reachability is pluggable beneath the RPC.** A private-mesh mode (self-hosted @@ -402,16 +406,16 @@ Tailscale is explicitly out of scope. ## 10. Error handling & lifecycle -| Failure | Detection | Behavior | -|---------|-----------|----------| -| Worker disconnects (crash / network) | `Attach` stream closes | Presence record removed from pool; worker reconnects and re-registers. In-flight `Exec` fails; leaf retry re-leases a healthy sandbox. | -| Command redelivered after reconnect | duplicate `req_id` at worker | Dedup cache re-emits cached `End`; else re-run (at-least-once). Partial-write risk documented (§8). | -| Relay restart | all parked streams drop | Workers reconnect; in-flight execs fail → leaf retry. No mid-exec durability. | -| Worker never connects / gone | no live stream for `sandbox_id`; harness deadline | Pool entry absent or evicted; `Exec` rejects; leaf retry re-leases elsewhere. | -| In-flight exec exceeds timeout | dual deadline (§8) | Worker SIGKILLs local child; harness synthesizes timeout — never hangs on a silent worker. | -| Output cap exceeded (poisoned/runaway) | harness byte counter on `ExecEvent` | Harness `Abort`s, truncates, surfaces `[output truncated]` to Pi. | -| Abort races with `End` | `req_id` correlation | Late `End` for an aborted `req_id` dropped; `Abort` for an ended `req_id` is a no-op. | -| Bad / missing token | relay validates on `Attach` | Stream rejected before it is parked; no pool entry created. | +| Failure | Detection | Behavior | +| -------------------------------------- | ------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------- | +| Worker disconnects (crash / network) | `Attach` stream closes | Presence record removed from pool; worker reconnects and re-registers. In-flight `Exec` fails; leaf retry re-leases a healthy sandbox. | +| Command redelivered after reconnect | duplicate `req_id` at worker | Dedup cache re-emits cached `End`; else re-run (at-least-once). Partial-write risk documented (§8). | +| Relay restart | all parked streams drop | Workers reconnect; in-flight execs fail → leaf retry. No mid-exec durability. | +| Worker never connects / gone | no live stream for `sandbox_id`; harness deadline | Pool entry absent or evicted; `Exec` rejects; leaf retry re-leases elsewhere. | +| In-flight exec exceeds timeout | dual deadline (§8) | Worker SIGKILLs local child; harness synthesizes timeout — never hangs on a silent worker. | +| Output cap exceeded (poisoned/runaway) | harness byte counter on `ExecEvent` | Harness `Abort`s, truncates, surfaces `[output truncated]` to Pi. | +| Abort races with `End` | `req_id` correlation | Late `End` for an aborted `req_id` dropped; `Abort` for an ended `req_id` is a no-op. | +| Bad / missing token | relay validates on `Attach` | Stream rejected before it is parked; no pool entry created. | **Lifecycle.** The `GrpcRelayTransport` is created at lease time; `close()` on leaf completion stops reading the `ExecEvent` stream. The worker's lifetime is independent — it @@ -420,12 +424,12 @@ shared-pool model. ## 11. Testing -| Layer | What | How | -|-------|------|-----| -| Pure unit | Frame reassembly, dedup cache, output-cap counter, timeout math, transport selection | Plain vitest / Go test, no I/O. | -| Contract | `GrpcRelayTransport` + Go worker against a real relay, worker pointed at a local bash | Round-trip every op (read/write/bash/grep), abort mid-stream, timeout, reconnect→dedup, output-cap. Core suite. | -| Conformance | The *same* battery run against **both** `KubectlTransport` and `GrpcRelayTransport` | One shared spec proving identical `SandboxTransport` contract — this is what makes them safely swappable (driver #2). | -| Live gate | Real Go worker pod dialing the in-cluster relay over `:443`; one leaf end-to-end on Kind, then OCP | Follows the existing leaf-smoke pattern; manifest-shape vitest parses the relay + worker Deployment YAML directly (no kustomize in CI). | +| Layer | What | How | +| ----------- | -------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------- | +| Pure unit | Frame reassembly, dedup cache, output-cap counter, timeout math, transport selection | Plain vitest / Go test, no I/O. | +| Contract | `GrpcRelayTransport` + Go worker against a real relay, worker pointed at a local bash | Round-trip every op (read/write/bash/grep), abort mid-stream, timeout, reconnect→dedup, output-cap. Core suite. | +| Conformance | The _same_ battery run against **both** `KubectlTransport` and `GrpcRelayTransport` | One shared spec proving identical `SandboxTransport` contract — this is what makes them safely swappable (driver #2). | +| Live gate | Real Go worker pod dialing the in-cluster relay over `:443`; one leaf end-to-end on Kind, then OCP | Follows the existing leaf-smoke pattern; manifest-shape vitest parses the relay + worker Deployment YAML directly (no kustomize in CI). | ## 12. Why this supersedes the Redis-Streams design @@ -439,7 +443,7 @@ that limited reach and portability: blocked by egress firewalls that allow only `:443`, and it forced a Redis availability dependency into the exec path. -This design keeps the outbound-dial insight and the frame *semantics* (`req_id`, dedup, +This design keeps the outbound-dial insight and the frame _semantics_ (`req_id`, dedup, dual timeout, output cap — §8) **verbatim**, and changes only the encoding and transport: protobuf over one gRPC bidi stream on `:443`. What survives unchanged from the earlier work is the `SandboxTransport` interface and the `KubectlTransport` rename (a pure, @@ -486,4 +490,4 @@ single-replica relay + the Go worker + the presence mirror. --- -*Assisted-By: Claude (Anthropic AI) * +_Assisted-By: Claude (Anthropic AI) _ diff --git a/docs/specs/2026-07-10-authbridge-egress-control-plane-poc-design.md b/docs/specs/2026-07-10-authbridge-egress-control-plane-poc-design.md index 4a4babd..c567d82 100644 --- a/docs/specs/2026-07-10-authbridge-egress-control-plane-poc-design.md +++ b/docs/specs/2026-07-10-authbridge-egress-control-plane-poc-design.md @@ -47,19 +47,19 @@ Prove, end-to-end and single-tenant, that **Rosso Cortex / AuthBridge** can be t the zero-trust credential plane on the serverless harness — doing both **credential injection** and **action control** on the harness's HTTP egress hops, with the credential itself never held by any model-influenced workload. The PoC is a **reference slice** (approach B): every seam is the shape it would -harden into, even though the credential is static and the control plugin's *judge* is canned. +harden into, even though the credential is static and the control plugin's _judge_ is canned. This reframes the Phase-2 plane, written before AuthBridge was committed as the mechanism and before the `#89` SandboxTransport inversion, around those two facts. ## 2. The four target capabilities -| # | Capability | PoC disposition | Fidelity | -|---|-----------|-----------------|----------| -| 1 | LLM credential injection (harness→LLM) | **Real** — harness holds a placeholder; AuthBridge `token-broker` swaps in the real provider key from `static-broker` | Production-shaped seam, static cred | -| 2 | Sandbox egress credential injection | **Real** — sandbox holds a placeholder; AuthBridge forward-proxy swaps in the real token for one external API | Production-shaped seam, static cred | -| 3 | SPARC/IBAC control (both hops) | **Real pipeline, stubbed judge** — real AuthBridge plugin on each hop; `sparc-stub` returns allow/deny by a simple rule | Seam real, verdict canned | -| 4 | Bring-your-own sandbox | **Stretch** — remote worker + its own egress AuthBridge dials the relay; the same leaf completes via `GrpcRelayTransport` | Gated on the transport live gate (ST5) | +| # | Capability | PoC disposition | Fidelity | +| --- | -------------------------------------- | ------------------------------------------------------------------------------------------------------------------------- | -------------------------------------- | +| 1 | LLM credential injection (harness→LLM) | **Real** — harness holds a placeholder; AuthBridge `token-broker` swaps in the real provider key from `static-broker` | Production-shaped seam, static cred | +| 2 | Sandbox egress credential injection | **Real** — sandbox holds a placeholder; AuthBridge forward-proxy swaps in the real token for one external API | Production-shaped seam, static cred | +| 3 | SPARC/IBAC control (both hops) | **Real pipeline, stubbed judge** — real AuthBridge plugin on each hop; `sparc-stub` returns allow/deny by a simple rule | Seam real, verdict canned | +| 4 | Bring-your-own sandbox | **Stretch** — remote worker + its own egress AuthBridge dials the relay; the same leaf completes via `GrpcRelayTransport` | Gated on the transport live gate (ST5) | **Non-goals (explicitly out of scope):** Keycloak, SPIRE/SPIFFE, RFC 8693 token-exchange, per-user identity, multi-tenant, HA/multi-replica AuthBridge, a real SPARC reflection service. @@ -78,12 +78,12 @@ for the decision and trade-offs. `ClusterIP Service`; every harness pod routes to it via `ANTHROPIC_BASE_URL`. Reverse/egress proxy, single known destination, no baked CA. Plugin chain: `inference-parser` → `SPARC/IBAC (inference mode)` → `token-broker (inject)`. **Pre-dispatch** control gate. The harness holds only a placeholder bearer, - and Z2's default-deny egress is tightened so the harness pod may reach *only* this gateway (enforceable + and Z2's default-deny egress is tightened so the harness pod may reach _only_ this gateway (enforceable "no key and can't phone home"). - **AuthBridge #2 — sandbox→external egress forward-proxy. CO-LOCATED with each sandbox.** Forward proxy with a baked CA (TLS-terminating), shipping with the sandbox bundle — a sidecar for in-cluster sandboxes, part of the remote bundle for BYO. Plugin chain: `mcp-parser` → `SPARC (mcp mode)` → `token-broker - (inject)`. **Action-time** control + egress-injection gate. The sandbox's `HTTPS_PROXY` points here and +(inject)`. **Action-time** control + egress-injection gate. The sandbox's `HTTPS_PROXY` points here and its trust store carries the baked CA. - **`static-broker` (new, tiny).** Minimal HTTP service implementing the `token-broker` contract (`POST /sessions/token`, keyed by `X-Server-Url` → returns the configured real token). No interactive @@ -160,8 +160,8 @@ flowchart LR classDef ext fill:#dcfce7,stroke:#16a34a,color:#111 ``` -*Solid thin* = placeholder-carrying request; *thick* (`==>`) = the real-credential egress leg injected at -AuthBridge; *dotted* = plugin side-calls and the `#4` stretch command path (sandbox dials out to the relay). +_Solid thin_ = placeholder-carrying request; _thick_ (`==>`) = the real-credential egress leg injected at +AuthBridge; _dotted_ = plugin side-calls and the `#4` stretch command path (sandbox dials out to the relay). ## 4. Flows @@ -170,7 +170,7 @@ AuthBridge; *dotted* = plugin side-calls and the `#4` stretch command path (sand 1. Harness sends an inference request to `ANTHROPIC_BASE_URL` (= AuthBridge #1) with `Authorization: Bearer `. 2. `inference-parser` extracts model + proposed tool calls into request context. -3. `SPARC/IBAC (inference mode)` calls `sparc-stub` → allow/deny. **Deny** → block (`403`) *before* +3. `SPARC/IBAC (inference mode)` calls `sparc-stub` → allow/deny. **Deny** → block (`403`) _before_ injection; the request never reaches the provider and never receives a real key. **Allow** → continue. 4. `token-broker` calls `static-broker` (`POST /sessions/token`, `X-Server-Url: `), gets the real key, and **replaces** the `Authorization` header. @@ -189,14 +189,14 @@ AuthBridge; *dotted* = plugin side-calls and the `#4` stretch command path (sand The sandbox worker (bundled with its own AuthBridge #2 egress proxy) runs outside the harness cluster, opens one outbound `Attach` bidi stream to the relay, and registers presence in Redis -(`sh:sandbox:records`). The harness dispatches the *same* leaf via `GrpcRelayTransport`; Hop-2 injection + +(`sh:sandbox:records`). The harness dispatches the _same_ leaf via `GrpcRelayTransport`; Hop-2 injection + control run unchanged inside the remote bundle. Depends on the SandboxTransport live gate (ST5). ## 5. Demo scenarios (what we exec to prove each capability) - **Cap #1 (LLM inject):** `kubectl exec -- printenv` shows only the placeholder (no real key); a leaf run completes (real key injected at AB1); AB1 audit log records the injection. Deny case: a - configured denylisted tool call is blocked at AB1 with `403` *before* any key is fetched. + configured denylisted tool call is blocked at AB1 with `403` _before_ any key is fetched. - **Cap #2 (sandbox inject):** the sandbox holds only a placeholder; a sandbox-run command hits an external echo/API target that reports back the `Authorization` it received = the **real** token; the sandbox never had it. Deny case: a denylisted MCP `tools/call` is blocked at AB2 before egress. @@ -208,9 +208,9 @@ control run unchanged inside the remote bundle. Depends on the SandboxTransport ## 6. Error handling & fail policies - **`token-broker`** is **fail-closed**: no/invalid token on the request → `401`; if `static-broker` is - unreachable the injection fails and **egress fails closed** — which is a *feature* here (it proves the + unreachable the injection fails and **egress fails closed** — which is a _feature_ here (it proves the workload cannot reach the target without the injector). -- **IBAC** is **fail-closed** (deny on judge error); **SPARC** is **fail-open** (a grounding *quality* +- **IBAC** is **fail-closed** (deny on judge error); **SPARC** is **fail-open** (a grounding _quality_ gate, not an auth control) — the PoC keeps these native defaults and demonstrates the distinction. - **Ordering:** control plugin before `token-broker` on both hops, so a denied action never receives a real credential. @@ -221,27 +221,27 @@ control run unchanged inside the remote bundle. Depends on the SandboxTransport ## 7. Verdicts on existing Phase-2 specs -| Spec | Verdict | Rationale | -|------|---------|-----------| -| **Z1** identity spine | **Defer (unchanged)** | PoC is single-tenant/static; no per-caller identity. Note: the **shared** AB1 is precisely what forces Z1 (mTLS/SPIFFE on harness→gateway) when multi-tenant attribution is wanted. | -| **Z2** harness lock-down | **Revise (narrow H1)** | Its "harness egress is fixed-destination, needs no proxy" holds for creds *alone*; adding SPARC/IBAC **control** on the (still fixed-destination) LLM hop justifies a proxy there. L2–L5 unchanged; H6 "separate pod" reinforced by ADR-0025. Tighten the default-deny egress to allow only → AB1. | -| **Z3** inference injector | **Revise / mechanism superseded for PoC** | Z3's plain Go injector + explicit *reject-AuthBridge* (I1) was decided for **injection only**; once control plugins share the hop, AuthBridge is justified. Keep Z3's shared-pod placement, provider routing, strip-then-set, audit-only; **AB1 replaces the plain-Go-injector mechanism.** | -| **Z4** MCP code-mode | **Keep** | Unchanged; it is the substrate the Hop-2 `mcp` gate inspects. PoC touches only egress interception of MCP `tools/call`. | -| **Z5** generalized credentialed egress | **Revise (implement static slice)** | Hop 2 **is** a minimal static-cred slice of Z5 (forward-proxy + baked CA + swap); per-user / RFC 8693 / token-exchange stay deferred. Refutes any "egress apparatus unjustified" read — that was Z2's point about the **harness**, never the **sandbox**. | -| **Z6, Z7** | **Defer (untouched)** | Subagents / red-team validation out of PoC scope. | +| Spec | Verdict | Rationale | +| -------------------------------------- | ----------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| **Z1** identity spine | **Defer (unchanged)** | PoC is single-tenant/static; no per-caller identity. Note: the **shared** AB1 is precisely what forces Z1 (mTLS/SPIFFE on harness→gateway) when multi-tenant attribution is wanted. | +| **Z2** harness lock-down | **Revise (narrow H1)** | Its "harness egress is fixed-destination, needs no proxy" holds for creds _alone_; adding SPARC/IBAC **control** on the (still fixed-destination) LLM hop justifies a proxy there. L2–L5 unchanged; H6 "separate pod" reinforced by ADR-0025. Tighten the default-deny egress to allow only → AB1. | +| **Z3** inference injector | **Revise / mechanism superseded for PoC** | Z3's plain Go injector + explicit _reject-AuthBridge_ (I1) was decided for **injection only**; once control plugins share the hop, AuthBridge is justified. Keep Z3's shared-pod placement, provider routing, strip-then-set, audit-only; **AB1 replaces the plain-Go-injector mechanism.** | +| **Z4** MCP code-mode | **Keep** | Unchanged; it is the substrate the Hop-2 `mcp` gate inspects. PoC touches only egress interception of MCP `tools/call`. | +| **Z5** generalized credentialed egress | **Revise (implement static slice)** | Hop 2 **is** a minimal static-cred slice of Z5 (forward-proxy + baked CA + swap); per-user / RFC 8693 / token-exchange stay deferred. Refutes any "egress apparatus unjustified" read — that was Z2's point about the **harness**, never the **sandbox**. | +| **Z6, Z7** | **Defer (untouched)** | Subagents / red-team validation out of PoC scope. | ### Proposed merge: Z3 + Z5 → one "AuthBridge egress control-plane" pattern Z3 and Z5 are, mechanically, the **same thing**: an AuthBridge instance on an egress hop running `parser → control → inject`. They differ only by **deployment profile**: -| Profile | Hop | Deployment | Destination | CA | Chain | -|---------|-----|-----------|-------------|----|-------| -| **A — LLM gateway** | harness→LLM | shared `Deployment`+`Service` | single, known | none | `inference-parser → SPARC/IBAC → token-broker` | -| **B — sandbox egress** | sandbox→ext | per-sandbox, co-located | arbitrary | baked | `mcp-parser → SPARC → token-broker` | +| Profile | Hop | Deployment | Destination | CA | Chain | +| ---------------------- | ----------- | ----------------------------- | ------------- | ----- | ---------------------------------------------- | +| **A — LLM gateway** | harness→LLM | shared `Deployment`+`Service` | single, known | none | `inference-parser → SPARC/IBAC → token-broker` | +| **B — sandbox egress** | sandbox→ext | per-sandbox, co-located | arbitrary | baked | `mcp-parser → SPARC → token-broker` | -**Proposal:** this spec (RC1) becomes the umbrella for the *mechanism*; the **mechanism** sections of Z3 -and Z5 are marked *superseded by RC1*, while Z3/Z5 are retained as the **deployment-profile detail** and the +**Proposal:** this spec (RC1) becomes the umbrella for the _mechanism_; the **mechanism** sections of Z3 +and Z5 are marked _superseded by RC1_, while Z3/Z5 are retained as the **deployment-profile detail** and the home of the **deferred per-user / token-exchange** work. This removes the duplicated "how injection works" prose across two specs and gives one pattern with two profiles. (Recorded in the registry lineage section of [`README.md`](README.md); with RC1 now **Accepted** (2026-07-14), the mechanism-superseded @@ -253,18 +253,18 @@ Dependency-ordered; Kind-first, OCP as the final progression. Live runs are driv with a hard timeout (never inside a subagent), logs redirected to `$LOG_DIR` and analyzed in subagents, per the repo's context-budget rules. -| Phase | Deliverable | Gate | -|-------|-------------|------| -| **RC1-0** | Shared services: `static-broker` + `sparc-stub` + AuthBridge image/config plumbing (ConfigMaps per instance) | unit tests (broker returns configured token; stub allow/deny); manifest-shape vitest (parse YAML directly) | -| **RC1-1** | **Hop 1** — shared AB1 `Deployment`+`Service`; harness placeholder + `ANTHROPIC_BASE_URL`→AB1; `token-broker` (real key) + SPARC/IBAC `inference` (stub); Z2 default-deny egress → AB1 only | live Kind: harness secret-free; leaf completes; deny-case blocks pre-inject | -| **RC1-2** | **Hop 2** — per-sandbox AB2 sidecar on the **current `KubectlTransport` sandbox**; baked CA; `token-broker` (real token, one external API) + SPARC `mcp` (stub) | live Kind: sandbox secret-free; external target sees real token; mcp deny blocks | -| **RC1-3** (stretch) | **BYO (#4)** — remote worker + AB2 bundle dials the relay; same leaf via `GrpcRelayTransport` | **gated on ST5**; live Kind: identical verdict via both transports | -| **RC1-4** | OCP 4.20 progression of RC1-1/RC1-2 | live OCP: both hops green via Route | +| Phase | Deliverable | Gate | +| ------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------- | +| **RC1-0** | Shared services: `static-broker` + `sparc-stub` + AuthBridge image/config plumbing (ConfigMaps per instance) | unit tests (broker returns configured token; stub allow/deny); manifest-shape vitest (parse YAML directly) | +| **RC1-1** | **Hop 1** — shared AB1 `Deployment`+`Service`; harness placeholder + `ANTHROPIC_BASE_URL`→AB1; `token-broker` (real key) + SPARC/IBAC `inference` (stub); Z2 default-deny egress → AB1 only | live Kind: harness secret-free; leaf completes; deny-case blocks pre-inject | +| **RC1-2** | **Hop 2** — per-sandbox AB2 sidecar on the **current `KubectlTransport` sandbox**; baked CA; `token-broker` (real token, one external API) + SPARC `mcp` (stub) | live Kind: sandbox secret-free; external target sees real token; mcp deny blocks | +| **RC1-3** (stretch) | **BYO (#4)** — remote worker + AB2 bundle dials the relay; same leaf via `GrpcRelayTransport` | **gated on ST5**; live Kind: identical verdict via both transports | +| **RC1-4** | OCP 4.20 progression of RC1-1/RC1-2 | live OCP: both hops green via Route | ## 9. Testing & verification - **Unit:** `static-broker` (token lookup by `X-Server-Url`, fail-closed on miss); `sparc-stub` (rule - eval, allow/deny/observe). + eval, allow/deny/observe). - **Manifest-shape (vitest):** parse the AB1/AB2/`static-broker`/`sparc-stub` manifests + the tightened harness egress `NetworkPolicy` directly as YAML (no kustomize in CI), asserting shape — matching the existing harness-egress-policy test approach. @@ -294,4 +294,4 @@ per the repo's context-budget rules. --- -*Assisted-By: Claude (Anthropic AI) * +_Assisted-By: Claude (Anthropic AI) _ diff --git a/docs/specs/2026-08-20-multi-protocol-model-provider-design.md b/docs/specs/2026-08-20-multi-protocol-model-provider-design.md index a6f03ec..70477a9 100644 --- a/docs/specs/2026-08-20-multi-protocol-model-provider-design.md +++ b/docs/specs/2026-08-20-multi-protocol-model-provider-design.md @@ -40,11 +40,11 @@ is a small, bounded generalization of the harness wrapper — not per-model spec Three places hardcode the Anthropic protocol; each is a dispatch point in the new design. -| Seam | File | Coupling | -|---|---|---| -| Model synthesis | `harness/src/run-turn.ts` `synthesizeCustomModel()` | Returns `Model<"anthropic-messages">`, `provider:"anthropic"`, `baseUrl = ANTHROPIC_BASE_URL` | -| Tool-choice nudge | `harness/src/tool-choice-extension.ts` | Injects `tool_choice: { type: "auto" }` — the **Anthropic object form**. vLLM/OpenAI reject the object; they want the string `"auto"` | -| Gateway/auth transform | `harness/src/run-turn.ts` `applyModelGateway()` | Rewrites to Bearer auth + strips `x-api-key`; seeds `ANTHROPIC_API_KEY` — Anthropic-specific | +| Seam | File | Coupling | +| ---------------------- | --------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------- | +| Model synthesis | `harness/src/run-turn.ts` `synthesizeCustomModel()` | Returns `Model<"anthropic-messages">`, `provider:"anthropic"`, `baseUrl = ANTHROPIC_BASE_URL` | +| Tool-choice nudge | `harness/src/tool-choice-extension.ts` | Injects `tool_choice: { type: "auto" }` — the **Anthropic object form**. vLLM/OpenAI reject the object; they want the string `"auto"` | +| Gateway/auth transform | `harness/src/run-turn.ts` `applyModelGateway()` | Rewrites to Bearer auth + strips `x-api-key`; seeds `ANTHROPIC_API_KEY` — Anthropic-specific | Pi's substrate already supports the target: `pi-fork/packages/ai/src/providers/openai-completions.ts` `createClient()` spreads `model.headers` into `defaultHeaders` and uses `model.baseUrl`; and @@ -55,11 +55,11 @@ via a custom header and ignores the Bearer token). ## 3. The protocol landscape -| `SH_MODEL_API` | Pi `api` | Endpoint | Serves | -|---|---|---|---| -| `anthropic` (default) | `anthropic-messages` | `/v1/messages` | Direct Anthropic, LiteLLM (Anthropic-format) | -| `openai-completions` | `openai-completions` | `/v1/chat/completions` | RITS, vLLM, OpenAI, Azure, most OSS gateways | -| `openai-responses` | `openai-responses` | `/v1/responses` | OpenAI, some gateways (optional; low priority) | +| `SH_MODEL_API` | Pi `api` | Endpoint | Serves | +| --------------------- | -------------------- | ---------------------- | ---------------------------------------------- | +| `anthropic` (default) | `anthropic-messages` | `/v1/messages` | Direct Anthropic, LiteLLM (Anthropic-format) | +| `openai-completions` | `openai-completions` | `/v1/chat/completions` | RITS, vLLM, OpenAI, Azure, most OSS gateways | +| `openai-responses` | `openai-responses` | `/v1/responses` | OpenAI, some gateways (optional; low priority) | ## 4. Design @@ -71,14 +71,14 @@ versions of the two extensions. `SH_MODEL_CUSTOM=1` stays the master "custom endpoint" switch. A new selector chooses the protocol; absent, it defaults to `anthropic` (today's behavior). -| Env | Meaning | Default | -|---|---|---| -| `SH_MODEL_API` | `anthropic` \| `openai-completions` \| `openai-responses` | `anthropic` | -| `SH_MODEL` | served model id (used as `id` + `name`) | — (required) | -| `SH_MODEL_BASE_URL` | endpoint base URL | falls back to `ANTHROPIC_BASE_URL` (anthropic) / `OPENAI_BASE_URL` (openai\*) for back-compat | -| `SH_MODEL_HEADERS` | JSON object of extra request headers, e.g. `{"RITS_API_KEY":"${RITS_API_KEY}"}`. String values support `${VAR}` interpolation from env, so a secret value flows in via a secretKeyRef env (no inline literal) | `{}` | -| `SH_MODEL_AUTH` | `bearer` \| `custom-header` \| `none` — how the endpoint authenticates | `bearer` | -| `SH_MODEL_CONTEXT_WINDOW` / `SH_MODEL_MAX_TOKENS` / `SH_MODEL_PROVIDER` | existing knobs, unchanged | 131072 / 8192 / per-protocol | +| Env | Meaning | Default | +| ----------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------- | +| `SH_MODEL_API` | `anthropic` \| `openai-completions` \| `openai-responses` | `anthropic` | +| `SH_MODEL` | served model id (used as `id` + `name`) | — (required) | +| `SH_MODEL_BASE_URL` | endpoint base URL | falls back to `ANTHROPIC_BASE_URL` (anthropic) / `OPENAI_BASE_URL` (openai\*) for back-compat | +| `SH_MODEL_HEADERS` | JSON object of extra request headers, e.g. `{"RITS_API_KEY":"${RITS_API_KEY}"}`. String values support `${VAR}` interpolation from env, so a secret value flows in via a secretKeyRef env (no inline literal) | `{}` | +| `SH_MODEL_AUTH` | `bearer` \| `custom-header` \| `none` — how the endpoint authenticates | `bearer` | +| `SH_MODEL_CONTEXT_WINDOW` / `SH_MODEL_MAX_TOKENS` / `SH_MODEL_PROVIDER` | existing knobs, unchanged | 131072 / 8192 / per-protocol | Secrets are **never** inline in `SH_MODEL_HEADERS` in manifests — the header **value** comes from a `secretKeyRef` env indirection at the deployment layer (same pattern as `RITS_API_KEY` / @@ -117,6 +117,7 @@ const model: Model<"openai-completions"> = { ### 4.3 Protocol-aware `toolChoiceExtension` The nudge must emit the form the protocol accepts: + - `anthropic` → `tool_choice: { type: "auto" }` (object) — unchanged. - `openai-*` → `tool_choice: "auto"` (string). vLLM/OpenAI reject the object (`Invalid value for 'function': 'None'`). @@ -128,6 +129,7 @@ tools present and unset" guard, the first-request log) stays. `applyModelGateway` is Anthropic-gateway-specific (Bearer + strip `x-api-key` + seed `ANTHROPIC_API_KEY`). For `openai-*` it must **not** apply that rewrite. Behavior by `SH_MODEL_AUTH`: + - `bearer` — Pi sends `Authorization: Bearer ` (standard OpenAI/vLLM). - `custom-header` — the endpoint authenticates via a header in `SH_MODEL_HEADERS` (e.g. RITS's `RITS_API_KEY`); **strip the default `Authorization`** so the SDK Bearer isn't sent, but keep a @@ -156,6 +158,7 @@ needs it; not on the critical path. ## 6. Testing & verification gate ### 6.1 Unit (fast, vitest) — extend `harness/test/run-turn-model.test.ts` + - `SH_MODEL_API=openai-completions` → synthesized model has `api:"openai-completions"`, `provider:"openai"`, `baseUrl` from `SH_MODEL_BASE_URL`, `headers` parsed from `SH_MODEL_HEADERS`. - `custom-header` auth → default `Authorization` stripped, custom header present. @@ -166,6 +169,7 @@ needs it; not on the critical path. - Invalid `SH_MODEL_API` / missing base URL → clear error. ### 6.2 Live gate (cluster) + - **OpenAI-compat, tool-capable:** point `SH_MODEL_API=openai-completions` at a RITS/vLLM route with the tool-call parser (e.g. Qwen2.5-72B-Instruct or Kimi) and run a **tool-requiring** `/turn` ("use your bash tool to run `echo PONG`") → assert a **structured** tool call executes in the @@ -201,4 +205,4 @@ before committing it to a run.** --- -*Assisted-By: Claude Code* +_Assisted-By: Claude Code_ diff --git a/docs/specs/2026-08-25-async-prompt-dispatch-design.md b/docs/specs/2026-08-25-async-prompt-dispatch-design.md index 0bf32e1..5226db3 100644 --- a/docs/specs/2026-08-25-async-prompt-dispatch-design.md +++ b/docs/specs/2026-08-25-async-prompt-dispatch-design.md @@ -46,18 +46,18 @@ and share the turn-execution core with `/turn` rather than forking it. ## 2. Current state — the seams we extend -| Seam | Today | This slice | -|---|---|---| -| `LeafEnvelope` (`harness/src/run-leaf.ts`) | `kind?: "converge" \| "solve"`, `problemStatement?` | `+ "prompt"`, `+ prompt?: string` | -| `LeafResult` union | `done→verdict`, `paused→gate`, `aborted`, `solved→patch`, `failed→reason` | `+ { status:"responded"; text; usage? }` | -| `runLeaf` dispatch | `if (env.kind === "solve") return runSolveLeaf(...)` | `+ if (env.kind === "prompt") return runPromptLeaf(...)` | -| `runTurn` (`harness/src/run-turn.ts`) | one function: create-or-404, wire extensions, run, extract text | factor a shared `executeTurn` core (both callers use it); `TurnResult += usage?` | -| `LeafResultRecord` (`leaf-result-store.ts`) | `status ∈ {done,failed,aborted,paused,solved}`, `verdict/gate/reason/patch/usage` | `+ "responded"` status, `+ text: string \| null` | -| `toResultRecord` | branch per status | `+ responded` branch | -| `isRunEnvelope` (`knative-server/src/server.ts`) | `isLeafEnvelope(o) \|\| isSolveEnvelope(o)` | `+ \|\| isPromptEnvelope(o)` | -| `handleLeafStatus` | wire cases for done/solved/paused/failed | `+ responded` case → `{ status:"responded", text }` | -| `classify-outcome`, `/runs` route, KEDA `ScaledJob` | kind-agnostic | **no change** | -| `leaf-job-runner` (`processOne`) | returns the leaf status union | type-level only: `+ "responded"` in the return union + doc comment; no behavioral change (a terminal `responded` acks like `solved`) | +| Seam | Today | This slice | +| --------------------------------------------------- | --------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------ | +| `LeafEnvelope` (`harness/src/run-leaf.ts`) | `kind?: "converge" \| "solve"`, `problemStatement?` | `+ "prompt"`, `+ prompt?: string` | +| `LeafResult` union | `done→verdict`, `paused→gate`, `aborted`, `solved→patch`, `failed→reason` | `+ { status:"responded"; text; usage? }` | +| `runLeaf` dispatch | `if (env.kind === "solve") return runSolveLeaf(...)` | `+ if (env.kind === "prompt") return runPromptLeaf(...)` | +| `runTurn` (`harness/src/run-turn.ts`) | one function: create-or-404, wire extensions, run, extract text | factor a shared `executeTurn` core (both callers use it); `TurnResult += usage?` | +| `LeafResultRecord` (`leaf-result-store.ts`) | `status ∈ {done,failed,aborted,paused,solved}`, `verdict/gate/reason/patch/usage` | `+ "responded"` status, `+ text: string \| null` | +| `toResultRecord` | branch per status | `+ responded` branch | +| `isRunEnvelope` (`knative-server/src/server.ts`) | `isLeafEnvelope(o) \|\| isSolveEnvelope(o)` | `+ \|\| isPromptEnvelope(o)` | +| `handleLeafStatus` | wire cases for done/solved/paused/failed | `+ responded` case → `{ status:"responded", text }` | +| `classify-outcome`, `/runs` route, KEDA `ScaledJob` | kind-agnostic | **no change** | +| `leaf-job-runner` (`processOne`) | returns the leaf status union | type-level only: `+ "responded"` in the return union + doc comment; no behavioral change (a terminal `responded` acks like `solved`) | The invariant that shapes everything below: `LeafResult` is a discriminated union where **each `status` discriminant maps to exactly one payload field** (`done`→`verdict`, `solved`→`patch`, …). @@ -97,8 +97,9 @@ best-effort — a usage hiccup never fails an otherwise-`responded` leaf. ```ts export function isPromptEnvelope(o: any): boolean { - return o && typeof o.sessionId === "string" && o.kind === "prompt" - && typeof o.prompt === "string"; + return ( + o && typeof o.sessionId === 'string' && o.kind === 'prompt' && typeof o.prompt === 'string' + ); } export function isRunEnvelope(o: any): boolean { return isLeafEnvelope(o) || isSolveEnvelope(o) || isPromptEnvelope(o); @@ -110,13 +111,13 @@ export function isRunEnvelope(o: any): boolean { their kinds). `toResultRecord` gains one branch: ```ts -if (result.status === "responded") - return { ...base, status: "responded", text: result.text, usage: result.usage ?? null }; +if (result.status === 'responded') + return { ...base, status: 'responded', text: result.text, usage: result.usage ?? null }; ``` ### 3.2 Runner — share the `/turn` core, don't fork it -The cleanest way to give a prompt leaf **full `/turn` parity** is to run *the same code* `/turn` +The cleanest way to give a prompt leaf **full `/turn` parity** is to run _the same code_ `/turn` runs. `runTurn` today is a single function that (a) opens-or-404s a session, (b) wires `flushExtension` + `k8sSandboxExtension(resolveSandboxConfig(...))` + `checkpointExtension` + `toolChoiceExtension` (+ optional `budgetVoterExtension`), (c) resolves the model, (d) runs one @@ -129,8 +130,8 @@ interface ExecuteTurnInput { prompt: string; sessionId?: string; config?: TurnConfig; - createIfAbsent: boolean; // session-open policy — see below - selection?: ModelSelection; // pre-resolved model/provider (leaf precedence); default: resolveModelSelection(config) + createIfAbsent: boolean; // session-open policy — see below + selection?: ModelSelection; // pre-resolved model/provider (leaf precedence); default: resolveModelSelection(config) } async function executeTurn(input: ExecuteTurnInput): Promise; ``` @@ -149,7 +150,7 @@ callers see a superset. A usage hiccup leaves `usage` undefined and never fails **The create-or-resume wrinkle.** `runTurn` today throws `"no session in backend"` when a `sessionId` is given but no checkpoint exists — that is `/turn`'s deliberate `404` contract. But a -*fresh* prompt leaf's `sessionId` has no prior checkpoint, and must **create**. This is the one +_fresh_ prompt leaf's `sessionId` has no prior checkpoint, and must **create**. This is the one behavioral difference between the two callers, so it is the one parameter: Session-open policy (design-level; the concrete checkpoint-existence probe follows whatever @@ -163,7 +164,7 @@ Session-open policy (design-level; the concrete checkpoint-existence probe follo So `/turn` keeps its 404, and a prompt leaf gets create-or-resume — the same pattern `runSolveLeaf` already implements, giving prompt leaves free at-least-once **resumability** on the async queue. This -`createIfAbsent` flag is the *only* behavioral parameter distinguishing the two callers; everything +`createIfAbsent` flag is the _only_ behavioral parameter distinguishing the two callers; everything else in the core is shared verbatim. **`runPromptLeaf`.** A thin adapter that resolves the leaf-family model selection, runs the core, @@ -171,23 +172,30 @@ and maps `TurnResult → LeafResult`: ```ts async function runPromptLeaf(env, config, deps?): Promise { - if (!env.prompt) return { status: "failed", reason: "bad_inputs" }; + if (!env.prompt) return { status: 'failed', reason: 'bad_inputs' }; const selection = resolveModelSelection({ model: env.model ?? config?.model, provider: env.provider ?? config?.provider, }); - const exec = deps?.executeTurn ?? executeTurn; // injectable seam for unit tests - const r = await exec({ prompt: env.prompt, sessionId: env.sessionId, config, createIfAbsent: true, selection }); - if (r.stopReason === "aborted") return { status: "aborted" }; - if (r.stopReason === "error") return { status: "failed", reason: "error", message: r.errorMessage }; - return { status: "responded", text: r.response, usage: r.usage }; + const exec = deps?.executeTurn ?? executeTurn; // injectable seam for unit tests + const r = await exec({ + prompt: env.prompt, + sessionId: env.sessionId, + config, + createIfAbsent: true, + selection, + }); + if (r.stopReason === 'aborted') return { status: 'aborted' }; + if (r.stopReason === 'error') + return { status: 'failed', reason: 'error', message: r.errorMessage }; + return { status: 'responded', text: r.response, usage: r.usage }; } ``` Dispatch in `runLeaf`, alongside the solve line: ```ts -if (env.kind === "prompt") return runPromptLeaf(env, config, deps); +if (env.kind === 'prompt') return runPromptLeaf(env, config, deps); ``` **Model precedence** is the clean superset already used by solve: `env.model ?? config?.model → @@ -216,8 +224,10 @@ The only wire addition is one status case: ```ts // handleLeafStatus -if (record.status === "responded") - return res.writeHead(200, JSON_HEADERS).end(JSON.stringify({ status: "responded", text: record.text })); +if (record.status === 'responded') + return res + .writeHead(200, JSON_HEADERS) + .end(JSON.stringify({ status: 'responded', text: record.text })); ``` Full async lifecycle, end to end: @@ -295,4 +305,4 @@ existing dispatch/poll helpers — no new script. --- -*Assisted-By: Claude (Anthropic AI) * +_Assisted-By: Claude (Anthropic AI) _ diff --git a/docs/specs/2026-08-26-st4-go-reference-worker-design.md b/docs/specs/2026-08-26-st4-go-reference-worker-design.md index aed8b8b..af7205e 100644 --- a/docs/specs/2026-08-26-st4-go-reference-worker-design.md +++ b/docs/specs/2026-08-26-st4-go-reference-worker-design.md @@ -89,7 +89,7 @@ ways to return the same bytes. Reported on #87 so ST1's author can correct the p This does not contradict the "static binary, no runtime deps" requirement: the binary itself stays CGO-free with no dynamic dependencies. `bash` is a dependency of the -*commands*, not of the worker — and in production the binary drops into a sandbox image +_commands_, not of the worker — and in production the binary drops into a sandbox image that already has a shell. It is the two **standalone demo images** built here that need a shell-bearing base. @@ -99,16 +99,16 @@ must be verified empirically before choosing; the plan carries that as an explic ## 4. Decisions -| # | Decision | Rationale | -|---|---|---| -| D1 | Per-exec `bash -c` child; **no** persistent shell | Every harness command is self-contained (`cd 'cwd' && …`, `env K=v bash -c …`), so no state must persist. Decisive: `base64 -d > f` only terminates on stdin EOF, which a shared shell cannot give per-exec. | -| D2 | Timeout → `ExecError{"timeout:"}` | All three existing transports reject with exactly this string (`exec.ts:79`, `persistent-exec.ts:129`, and `GrpcRelayTransport`'s own deadline). The worker's timer normally fires first, so its choice is what the caller sees; `End{-1}` would make a timeout resolve as a success with partial output. | -| D3 | `streaming: false` → buffer, one `Chunk` + terminal | §3.2. | -| D4 | Dedup keyed `req_id` + fingerprint guard | §3.1. | -| D5 | Bounded concurrency, `capacity_max = N` (default 4) | Sandboxes are shared, so concurrent execs are real; strict serialization head-of-line blocks one leaf behind another's slow `bash`. §7's "one in flight" describes the ordering guarantee, not a worker cap. | -| D6 | Contract battery against an in-process **Go** fake relay; real TS relay behind `SH_LIVE_RELAY=1` | The battery's subjects are all worker-side (SIGKILL, process groups, dedup); the relay is a pure bridge and contributes nothing to them. Decisively, dedup can *only* be tested against a relay that misbehaves on purpose — the real one has no redelivery path. Keeps the existing CI Go job free of node/pnpm/redis. | -| D7 | Delete the HELLO WORLD path; rewrite its docs | Once bash runs, a mode that fabricates output is a liability in a component whose security story is "only executes commands and returns bytes". The demo becomes strictly better: the FLAGGED verdict comes from real file content. | -| D8 | Reconnect in-process, cache preserved | Today the worker returns on `recv` error and exits; a pod restart wipes the cache, making "reconnect → dedup" unreachable by construction. | +| # | Decision | Rationale | +| --- | ------------------------------------------------------------------------------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| D1 | Per-exec `bash -c` child; **no** persistent shell | Every harness command is self-contained (`cd 'cwd' && …`, `env K=v bash -c …`), so no state must persist. Decisive: `base64 -d > f` only terminates on stdin EOF, which a shared shell cannot give per-exec. | +| D2 | Timeout → `ExecError{"timeout:"}` | All three existing transports reject with exactly this string (`exec.ts:79`, `persistent-exec.ts:129`, and `GrpcRelayTransport`'s own deadline). The worker's timer normally fires first, so its choice is what the caller sees; `End{-1}` would make a timeout resolve as a success with partial output. | +| D3 | `streaming: false` → buffer, one `Chunk` + terminal | §3.2. | +| D4 | Dedup keyed `req_id` + fingerprint guard | §3.1. | +| D5 | Bounded concurrency, `capacity_max = N` (default 4) | Sandboxes are shared, so concurrent execs are real; strict serialization head-of-line blocks one leaf behind another's slow `bash`. §7's "one in flight" describes the ordering guarantee, not a worker cap. | +| D6 | Contract battery against an in-process **Go** fake relay; real TS relay behind `SH_LIVE_RELAY=1` | The battery's subjects are all worker-side (SIGKILL, process groups, dedup); the relay is a pure bridge and contributes nothing to them. Decisively, dedup can _only_ be tested against a relay that misbehaves on purpose — the real one has no redelivery path. Keeps the existing CI Go job free of node/pnpm/redis. | +| D7 | Delete the HELLO WORLD path; rewrite its docs | Once bash runs, a mode that fabricates output is a liability in a component whose security story is "only executes commands and returns bytes". The demo becomes strictly better: the FLAGGED verdict comes from real file content. | +| D8 | Reconnect in-process, cache preserved | Today the worker returns on `recv` error and exits; a pod restart wipes the cache, making "reconnect → dedup" unreachable by construction. | ## 5. Architecture @@ -165,13 +165,13 @@ concurrent `Send`. ### Terminal-frame mapping -| Outcome | Frame | -|---|---| -| child exited normally | `End{req_id, exit_code: N}` | -| killed by `Abort` | `End{req_id, exit_code: -1}` (signal/none) | -| `timeout_s` expired | `ExecError{req_id, "timeout:"}` | -| spawn / pipe failure | `ExecError{req_id, message}` | -| queue overflow | `ExecError{req_id, "busy: queue full"}` | +| Outcome | Frame | +| --------------------- | ------------------------------------------ | +| child exited normally | `End{req_id, exit_code: N}` | +| killed by `Abort` | `End{req_id, exit_code: -1}` (signal/none) | +| `timeout_s` expired | `ExecError{req_id, "timeout:"}` | +| spawn / pipe failure | `ExecError{req_id, message}` | +| queue overflow | `ExecError{req_id, "busy: queue full"}` | The cache stores whichever terminal frame was produced, not strictly an `End` — a superset of the acceptance wording, so a redelivered `req_id` whose first run timed out re-emits the @@ -200,14 +200,14 @@ still see EOF or anything reading stdin blocks forever. So: write `spec.Stdin` i then close unconditionally. **Two pipes, 32 KiB reads.** Independent goroutines drain stdout and stderr into 32 KiB -buffers; each read emits one `Chunk` on the matching `Stream` enum. The read size *is* the +buffers; each read emits one `Chunk` on the matching `Stream` enum. The read size _is_ the per-frame cap, satisfying §8 backpressure with no extra buffering layer. A `Sink.Chunk` error means the stream is gone: the runner cancels its context (killing the process group) and returns that error, and the session emits no terminal frame — there is nowhere to send it. -*Known property:* relative ordering **between** stdout and stderr is not preserved, since +_Known property:_ relative ordering **between** stdout and stderr is not preserved, since they are independent pipes. The harness does not depend on it — it collects stdout and replays both to `onData` — but interleaving is not a guarantee this worker makes. @@ -238,12 +238,12 @@ answered twice — once by the original's own terminal frame, once by the cache, completed-only lookup below cannot see while that window is still open. **Abort reaches queued execs.** An `inflight` map `req_id → cancel` is populated at -*enqueue*, not at start, so aborting a not-yet-started exec cancels its context and the +_enqueue_, not at start, so aborting a not-yet-started exec cancels its context and the pool drops it without spawning `bash`. `Abort` for an unknown `req_id` is a no-op (§8). **On disconnect, in-flight execs are cancelled.** Their output has nowhere to go and the relay has already failed them harness-side (`relay.ts:89-93`); keeping the children alive -only orphans work. *Consequence, stated plainly:* a killed-in-flight exec leaves no cached +only orphans work. _Consequence, stated plainly:_ a killed-in-flight exec leaves no cached terminal frame, so a redelivery re-runs it. Dedup protects **completed** execs only — which is precisely at-least-once, and honest about it. @@ -256,24 +256,24 @@ of today's hardcoded `["hello-world"]`. Bounded LRU, 256 entries, `req_id → {fingerprint, terminal frame}` where fingerprint is SHA-256 over `command` + `stdin`. -| Case | Behavior | -|---|---| -| unknown `req_id` | run it | -| known `req_id`, fingerprint matches | re-emit cached terminal frame, do **not** run | -| known `req_id`, fingerprint differs, original completed | run it fresh, log a warning (§3.1 collision) | +| Case | Behavior | +| ------------------------------------------------------------- | --------------------------------------------------------------------------------------------------- | +| unknown `req_id` | run it | +| known `req_id`, fingerprint matches | re-emit cached terminal frame, do **not** run | +| known `req_id`, fingerprint differs, original completed | run it fresh, log a warning (§3.1 collision) | | known `req_id`, fingerprint differs, original still in flight | refuse it with `ExecError`, log a warning — running it would mean two concurrent execs under one id | ### 6.4 `cmd/worker` — wiring -| Env var | Default | Feeds | -|---|---|---| -| `RELAY_ADDR` | `localhost:8443` | dial target | -| `SANDBOX_ID` | `sbx-laptop-1` | `Hello.sandbox_id` | -| `SANDBOX_TOKEN` | `dev-token` | `authorization: Bearer …` | -| `RELAY_TLS` | `0` | TLS vs h2c | -| `WORKER_MAX_CONCURRENT` | `4` | pool size N, `Hello.capacity_max` | -| `SANDBOX_IMAGE` | `""` | `Hello.image` | -| `SANDBOX_TRUST` | `untrusted` | `Hello.trust` | +| Env var | Default | Feeds | +| ----------------------- | ---------------- | --------------------------------- | +| `RELAY_ADDR` | `localhost:8443` | dial target | +| `SANDBOX_ID` | `sbx-laptop-1` | `Hello.sandbox_id` | +| `SANDBOX_TOKEN` | `dev-token` | `authorization: Bearer …` | +| `RELAY_TLS` | `0` | TLS vs h2c | +| `WORKER_MAX_CONCURRENT` | `4` | pool size N, `Hello.capacity_max` | +| `SANDBOX_IMAGE` | `""` | `Hello.image` | +| `SANDBOX_TRUST` | `untrusted` | `Hello.trust` | `Hello.arch` comes from `runtime.GOARCH`, `Hello.capabilities` from `exec.LookPath` (§6.2). `Hello.labels` is left empty: nothing consumes it yet (`relay.ts:74` defers `capacityMax` @@ -303,14 +303,14 @@ one chunk per stream. **Contract — real runner, real gRPC, `relaytest`** — the acceptance battery: -| Item | Shape | -|---|---| -| read | `cat f` → stdout bytes, `End{0}` | -| write | `base64 -d > f` + stdin → content on disk, `End{0}` | -| bash | `echo hi; echo oops >&2; exit 7` → STDOUT chunk + STDERR chunk + `End{7}` | -| grep | `rg pat f` (falls back to `grep` when `rg` is absent) → multiple STDOUT chunks | -| abort mid-stream | emitter, `Abort` after chunk 1 → `End{-1}`, no further chunks, group gone | -| timeout | `sleep 30`, `timeout_s: 1` → `ExecError{"timeout:1"}` | +| Item | Shape | +| ----------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| read | `cat f` → stdout bytes, `End{0}` | +| write | `base64 -d > f` + stdin → content on disk, `End{0}` | +| bash | `echo hi; echo oops >&2; exit 7` → STDOUT chunk + STDERR chunk + `End{7}` | +| grep | `rg pat f` (falls back to `grep` when `rg` is absent) → multiple STDOUT chunks | +| abort mid-stream | emitter, `Abort` after chunk 1 → `End{-1}`, no further chunks, group gone | +| timeout | `sleep 30`, `timeout_s: 1` → `ExecError{"timeout:1"}` | | reconnect → dedup | `req_id 4` runs `echo x >> log` and completes; drop the stream; re-Attach; resend `req_id 4` → cached `End` re-emitted **and `log` still has one line** — the marker is what proves "no re-run" rather than merely "same frame" | **Gated live** — `SH_LIVE_RELAY=1` runs read/write/bash/grep/abort against the real TS relay @@ -342,9 +342,9 @@ covered by the fallback. ## 10. Risks -| Risk | Mitigation | -|---|---| -| Fake relay encodes *my reading* of the contract, not the relay's behavior | The `SH_LIVE_RELAY=1` tier runs the same battery against the real relay; ST5 (#88) is the true interop gate. | -| `req_id` collision persists until ST1/ST3 fix it | Fingerprint guard converts a silent wrong result into a correct re-run plus a warning (§3.1). | -| Worker runs arbitrary commands — that is the job | No credentials and no orchestration in the worker (§7); bounded concurrency and queue caps limit resource exhaustion; trust boundary is the sandbox itself, unchanged. | -| `ubi-micro` may lack `bash`, blocking the demo image | Verified empirically as an explicit plan step before choosing the base (§3.3). | +| Risk | Mitigation | +| ------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| Fake relay encodes _my reading_ of the contract, not the relay's behavior | The `SH_LIVE_RELAY=1` tier runs the same battery against the real relay; ST5 (#88) is the true interop gate. | +| `req_id` collision persists until ST1/ST3 fix it | Fingerprint guard converts a silent wrong result into a correct re-run plus a warning (§3.1). | +| Worker runs arbitrary commands — that is the job | No credentials and no orchestration in the worker (§7); bounded concurrency and queue caps limit resource exhaustion; trust boundary is the sandbox itself, unchanged. | +| `ubi-micro` may lack `bash`, blocking the demo image | Verified empirically as an explicit plan step before choosing the base (§3.3). | diff --git a/docs/specs/2026-08-26-turn-sse-streaming-design.md b/docs/specs/2026-08-26-turn-sse-streaming-design.md index 9a422be..5d71a2e 100644 --- a/docs/specs/2026-08-26-turn-sse-streaming-design.md +++ b/docs/specs/2026-08-26-turn-sse-streaming-design.md @@ -11,12 +11,12 @@ Builds on (reuse, no redesign): the shared `executeTurn` turn core and its exten [`2026-06-17-m4-knative-serverless-wrapper-design.md`](2026-06-17-m4-knative-serverless-wrapper-design.md)), Pi's session event surface (`pi.on(...)`), and the Knative HTTP entrypoint's existing route wiring. -> **What this slice is NOT.** Not a new route — streaming is a *representation* of `/turn`, chosen by +> **What this slice is NOT.** Not a new route — streaming is a _representation_ of `/turn`, chosen by > `Accept`, not a `/turn/stream` alias. Not the async "fire-and-poll" path: that is the companion > `kind:"prompt"` leaf ([#168](https://github.com/rossoctl/serverless-harness/issues/168), > [ADR-0028](../adrs/0028-async-prompt-dispatch.md)) — orthogonal ("watch live" vs. "background and > poll"). Not an auth/credential change. Not a new turn engine: streaming and non-streaming run the -> **same** `executeTurn`; the only new thing is an event *sink* and its SSE serialization. +> **same** `executeTurn`; the only new thing is an event _sink_ and its SSE serialization. --- @@ -44,14 +44,14 @@ share one turn engine rather than fork it. ## 2. Current state — the seams we extend -| Seam | Today | This slice | -|---|---|---| -| `handleTurn` (`packages/knative-server/src/server.ts`) | read body → parse → validate `prompt` → `runTurn(...)` → `res.writeHead(200, JSON_HEADERS).end(JSON.stringify(result))` | branch on `Accept` **after** validation; the sync emission line is untouched | -| `executeTurn` (`harness/src/run-turn.ts`) | builds `extensionFactories`, opens/creates session, `session.prompt`, extracts text, returns `TurnResult` | `ExecuteTurnInput += onEvent?`, `signal?` (both optional, additive); when present, push `sseExtension` and wire abort | -| `TurnResult` | `{ sessionId; response; stopReason; errorMessage?; usage? }` | **unchanged** — the terminal SSE frame is derived from it | -| Pi session events (`pi.on`) | consumed by `flushExtension`, `checkpointExtension`, … | a new `sseExtension` consumes `message_update` / `tool_execution_start` / `tool_execution_end` | -| `session.abort()` (`pi-fork/.../agent-session.ts`) | invoked on shutdown paths | invoked on client disconnect via an `AbortSignal` | -| `/turn` route guard (`server.ts`) | wrapper writes `500` only `if (!res.headersSent)` | already correct for streaming — a post-flush throw can't overwrite status | +| Seam | Today | This slice | +| ------------------------------------------------------ | ----------------------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------- | +| `handleTurn` (`packages/knative-server/src/server.ts`) | read body → parse → validate `prompt` → `runTurn(...)` → `res.writeHead(200, JSON_HEADERS).end(JSON.stringify(result))` | branch on `Accept` **after** validation; the sync emission line is untouched | +| `executeTurn` (`harness/src/run-turn.ts`) | builds `extensionFactories`, opens/creates session, `session.prompt`, extracts text, returns `TurnResult` | `ExecuteTurnInput += onEvent?`, `signal?` (both optional, additive); when present, push `sseExtension` and wire abort | +| `TurnResult` | `{ sessionId; response; stopReason; errorMessage?; usage? }` | **unchanged** — the terminal SSE frame is derived from it | +| Pi session events (`pi.on`) | consumed by `flushExtension`, `checkpointExtension`, … | a new `sseExtension` consumes `message_update` / `tool_execution_start` / `tool_execution_end` | +| `session.abort()` (`pi-fork/.../agent-session.ts`) | invoked on shutdown paths | invoked on client disconnect via an `AbortSignal` | +| `/turn` route guard (`server.ts`) | wrapper writes `500` only `if (!res.headersSent)` | already correct for streaming — a post-flush throw can't overwrite status | The invariant that shapes everything below: **the turn core emits neutral domain frames; the server owns the SSE transport.** `runTurn`/`executeTurn` never learn what HTTP or SSE is; they call an @@ -72,12 +72,18 @@ transport, plus the Pi→frame translator. ```ts export type TurnStreamFrame = - | { type: "text"; delta: string } // assistant-text token - | { type: "thinking"; delta: string } // reasoning token (optional; see §3.5) - | { type: "tool_use"; id: string; name: string; args: unknown } // tool call started (args verbatim) - | { type: "tool_result"; id: string; isError: boolean; preview: string } // tool call ended (clipped) - | { type: "done"; sessionId: string; stopReason: string; usage?: LeafUsage } - | { type: "error"; sessionId: string; stopReason: string; errorMessage?: string; usage?: LeafUsage }; + | { type: 'text'; delta: string } // assistant-text token + | { type: 'thinking'; delta: string } // reasoning token (optional; see §3.5) + | { type: 'tool_use'; id: string; name: string; args: unknown } // tool call started (args verbatim) + | { type: 'tool_result'; id: string; isError: boolean; preview: string } // tool call ended (clipped) + | { type: 'done'; sessionId: string; stopReason: string; usage?: LeafUsage } + | { + type: 'error'; + sessionId: string; + stopReason: string; + errorMessage?: string; + usage?: LeafUsage; + }; ``` **Tool-payload fidelity is level "B"**: `tool_use` carries the tool `name` and its `args` verbatim @@ -92,22 +98,35 @@ untruncated result. `flushExtension` — that registers handlers and translates each Pi event into a frame: ```ts -export function sseExtension(onEvent: (f: TurnStreamFrame) => void, opts?: { previewBytes?: number }): ExtensionFactory { +export function sseExtension( + onEvent: (f: TurnStreamFrame) => void, + opts?: { previewBytes?: number }, +): ExtensionFactory { return (pi) => { - pi.on("message_update", (e) => { + pi.on('message_update', (e) => { const a = e.assistantMessageEvent; - if (a.type === "text_delta" && a.delta) onEvent({ type: "text", delta: a.delta }); - else if (a.type === "thinking_delta" && a.delta) onEvent({ type: "thinking", delta: a.delta }); + if (a.type === 'text_delta' && a.delta) onEvent({ type: 'text', delta: a.delta }); + else if (a.type === 'thinking_delta' && a.delta) + onEvent({ type: 'thinking', delta: a.delta }); }); - pi.on("tool_execution_start", (e) => onEvent({ type: "tool_use", id: e.toolCallId, name: e.toolName, args: e.args })); - pi.on("tool_execution_end", (e) => onEvent({ type: "tool_result", id: e.toolCallId, isError: e.isError, preview: clip(e.result, opts?.previewBytes) })); + pi.on('tool_execution_start', (e) => + onEvent({ type: 'tool_use', id: e.toolCallId, name: e.toolName, args: e.args }), + ); + pi.on('tool_execution_end', (e) => + onEvent({ + type: 'tool_result', + id: e.toolCallId, + isError: e.isError, + preview: clip(e.result, opts?.previewBytes), + }), + ); }; } ``` `sseExtension` emits **only incremental progress frames** (`text`/`thinking`/`tool_use`/ `tool_result`). Terminal frames (`done`/`error`) are the server's job (§3.3, §3.4), derived from the -`TurnResult` the core returns — so the streamed client ends with the *same facts* a sync client reads. +`TurnResult` the core returns — so the streamed client ends with the _same facts_ a sync client reads. ### 3.2 Turn core — two additive optional inputs @@ -117,8 +136,8 @@ returns the same `TurnResult` and throws the same errors: ```ts interface ExecuteTurnInput { // ...existing: prompt, sessionId, config, createIfAbsent, selection... - onEvent?: (frame: TurnStreamFrame) => void; // present ⇒ append sseExtension(onEvent) to extensionFactories - signal?: AbortSignal; // present ⇒ signal.onabort → session.abort() + onEvent?: (frame: TurnStreamFrame) => void; // present ⇒ append sseExtension(onEvent) to extensionFactories + signal?: AbortSignal; // present ⇒ signal.onabort → session.abort() } ``` @@ -135,24 +154,24 @@ The branch lives inside `handleTurn`, **after** its unchanged front matter (body `prompt` validation — all 400 paths preserved), right before the `runTurn` call: ```ts -const wantsStream = /text\/event-stream/i.test(req.headers.accept ?? ""); +const wantsStream = /text\/event-stream/i.test(req.headers.accept ?? ''); if (wantsStream) return handleTurnStream(prompt, sessionId, req, res); // unchanged sync path: runTurn(...) → res.writeHead(200, JSON_HEADERS).end(JSON.stringify(result)) ``` -`handleTurnStream` is a new sibling in `server.ts` (transport lives with the server; only the *frame -types* are imported from `@sh/harness/turn-stream`). Its shape: +`handleTurnStream` is a new sibling in `server.ts` (transport lives with the server; only the _frame +types_ are imported from `@sh/harness/turn-stream`). Its shape: - **Lazy header flush.** It does **not** write the `200` on entry. A single private `writeFrame(res, - frame)` helper flushes the SSE headers on the **first** frame and serializes every frame to the SSE +frame)` helper flushes the SSE headers on the **first** frame and serializes every frame to the SSE wire form `event: \ndata: \n\n`. Named events (not bare `data:`) so `curl -N` shows `event: text` / `event: tool_use` and `EventSource` clients can `addEventListener` per type. The lazy flush is what preserves status-code parity for pre-turn failures (§3.4). - **SSE headers** (written on first frame): `Content-Type: text/event-stream`, `Cache-Control: - no-cache`, `Connection: keep-alive`, `X-Accel-Buffering: no`. No `Content-Length` — Node emits +no-cache`, `Connection: keep-alive`, `X-Accel-Buffering: no`. No `Content-Length` — Node emits chunked transfer encoding and flushes each frame. - **The call:** `executeTurn({ prompt, sessionId, config: buildConfig(), createIfAbsent: false, - onEvent: (f) => writeFrame(res, f), signal: ac.signal })`. Progress frames stream as they arrive. +onEvent: (f) => writeFrame(res, f), signal: ac.signal })`. Progress frames stream as they arrive. - **Terminal frame:** on resolve, derive from the returned `TurnResult` — `done` for a clean `stopReason`, `error` for `error`/`aborted` — carrying every `TurnResult` field, then `res.end()`. - **Heartbeat:** a timer emits an SSE comment (`: keepalive\n\n`) every @@ -165,7 +184,7 @@ types* are imported from `@sh/harness/turn-stream`). Its shape: Lazy header flush partitions failures into three regimes, each mapped to preserve the sync contract: 1. **Pre-flight (before the branch).** Bad JSON, missing `prompt` — handled in `handleTurn`'s front - matter *before* `handleTurnStream` is called, so they stay real `400`s with the identical body. No + matter _before_ `handleTurnStream` is called, so they stay real `400`s with the identical body. No divergence. 2. **Pre-first-frame (session open fails).** `createIfAbsent:false` with an unknown `sessionId` throws `"no session in backend"` at session-open, before any progress frame. Since `writeFrame` @@ -177,23 +196,24 @@ Lazy header flush partitions failures into three regimes, each mapped to preserv codes are spent. The catch emits a terminal `error` frame then `res.end()`s. Guarded by `!res.writableEnded` so a concurrent disconnect can't double-write or `EPIPE`. -**Terminal frame selection.** When `executeTurn` *returns* a `TurnResult` (the HTTP-200 equivalent), +**Terminal frame selection.** When `executeTurn` _returns_ a `TurnResult` (the HTTP-200 equivalent), the terminal frame carries **every field the sync `TurnResult` exposes** (`sessionId`, `stopReason`, `usage`, and `errorMessage` when present). The only difference between `done` and `error` is the -*event name*, chosen by whether `stopReason` is a clean finish (`end_turn`/`max_tokens` → `done`; +_event name_, chosen by whether `stopReason` is a clean finish (`end_turn`/`max_tokens` → `done`; `error`/`aborted` → `error`). So a model that ends in an error stop-reason surfaces as an `error` frame with the same `errorMessage` a sync caller would read — the frame name is pure sugar over the same facts, and resume parity holds because `sessionId`/`stopReason` are always present. **The guarantee, stated plainly:** the non-streaming response is unchanged because its emission line -is never edited, *and* every pre-commit failure a sync caller could hit still returns the same status -+ body on a streaming request. Streaming only *adds* a post-commit `error`-frame surface that has no -sync equivalent (status codes aren't available after the first byte). +is never edited, _and_ every pre-commit failure a sync caller could hit still returns the same status + +- body on a streaming request. Streaming only _adds_ a post-commit `error`-frame surface that has no + sync equivalent (status codes aren't available after the first byte). ### 3.5 Thinking frames are optional `thinking` deltas (from `thinking_delta`) are surfaced as their own event so a watcher can render or -ignore reasoning independently — the distinct event type *is* the opt-out, so no gating query param is +ignore reasoning independently — the distinct event type _is_ the opt-out, so no gating query param is needed. But the frame is **best-effort and may never fire**: synthesized custom models configured `reasoning:false`, and some OpenAI-compatible endpoints, emit no thinking stream. Clients MUST treat `thinking` (and indeed every progress frame type) as optional and absence as normal. @@ -225,12 +245,12 @@ all deltas to the end and defeat the feature. Two responses, both part of this d - **Mitigation:** annotate the streaming revision with `autoscaling.knative.dev/target-burst-capacity: "0"`, which drops the activator from the path once the pod is up, giving a direct Kourier→pod stream. -- **Validation:** the gated smoke (§5) asserts *inter-frame arrival timing*, not just final content — +- **Validation:** the gated smoke (§5) asserts _inter-frame arrival timing_, not just final content — if the activator buffers, deltas arrive clumped and the test fails loudly. "Streaming actually streams end-to-end" is proven against a deployed revision, never assumed. **Timeout ceiling — no new knob.** A streamed turn is bounded by the revision's `timeoutSeconds` -(default 300s) exactly as the sync `/turn` already is; streaming makes that bound *visible* (frames +(default 300s) exactly as the sync `/turn` already is; streaming makes that bound _visible_ (frames until cutoff) rather than worse. Long-horizon work is the async path's job (#168), not this one. --- @@ -254,7 +274,7 @@ until cutoff) rather than worse. Long-horizon work is the async path's job (#168 - No/other `Accept` → **golden byte-for-byte** assertion against the current JSON response. If a future edit perturbs the sync bytes, this fails. - `Accept: text/event-stream` → `Content-Type: text/event-stream`, ordered frames, terminal `done`. - - Bad `sessionId` + streaming `Accept` → real **404 JSON** (pre-first-frame regime), *not* an error + - Bad `sessionId` + streaming `Accept` → real **404 JSON** (pre-first-frame regime), _not_ an error frame. - Missing `prompt` + streaming `Accept` → **400** (pre-flight). - **Disconnect/abort:** destroy the client socket mid-stream → assert the `AbortSignal` handed to @@ -272,11 +292,11 @@ parity**. ### 5.4 Acceptance-criteria coverage (issue #167) -| Criterion | Covered by | -|---|---| -| Default `/turn` unchanged | §5.2 golden byte-for-byte test | -| Session persisted/resumable identically | §5.3 live-smoke follow-up turn (real engine) | -| Client disconnect aborts the turn | §5.2 disconnect/abort | +| Criterion | Covered by | +| ------------------------------------------ | ---------------------------------------------------------- | +| Default `/turn` unchanged | §5.2 golden byte-for-byte test | +| Session persisted/resumable identically | §5.3 live-smoke follow-up turn (real engine) | +| Client disconnect aborts the turn | §5.2 disconnect/abort | | `curl -N` example + smoke asserting deltas | §5.3 + a documented `curl -N` example in the endpoint docs | --- @@ -306,4 +326,4 @@ parity**. --- -*Assisted-By: Claude (Anthropic AI) * +_Assisted-By: Claude (Anthropic AI) _ diff --git a/docs/specs/2026-08-30-seam-output-cap-truncation-design.md b/docs/specs/2026-08-30-seam-output-cap-truncation-design.md index ac26250..c674299 100644 --- a/docs/specs/2026-08-30-seam-output-cap-truncation-design.md +++ b/docs/specs/2026-08-30-seam-output-cap-truncation-design.md @@ -15,7 +15,7 @@ guarantee does not hold, and the three ways it fails were filed separately durin whole-branch review that closed the epic (#184). **Truncation has no representation in the seam's return type.** `ExecInPod` resolves -`{ stdout, exitCode: number | null }`, and `exitCode === null` is overloaded to mean *both* +`{ stdout, exitCode: number | null }`, and `exitCode === null` is overloaded to mean _both_ "our cap tripped" and "the process produced no status" (signalled; gRPC `end.exitCode < 0`; stream end without an `End` frame). Callers cannot distinguish them, so every consequence below is a consequence of one missing bit. @@ -31,7 +31,7 @@ below is a consequence of one missing bit. channel and 20 MiB comes back; the same read through either per-call transport returns 8 MiB plus `[output truncated]`. Pi can tell the backends apart, which is what the guarantee exists to prevent. -- **#185 — the cap's *enforcement* is not equivalent, and the battery cannot see it.** +- **#185 — the cap's _enforcement_ is not equivalent, and the battery cannot see it.** `GrpcRelayTransport` issues `Abort`, which kills the remote process. `KubectlTransport` kills only the local `kubectl` client; the in-pod process stops on EPIPE, if at all. The shared battery reduces each mechanism to a `producerStopped()` boolean supplied by that @@ -52,8 +52,8 @@ change: one behavioural fix, five message fixes, and one new cap. **Capping the read path costs more than #180 states.** `pi-fork/.../tools/read.ts:277` calls `ops.readFile(absolutePath)` for the **whole file** and only then slices by `offset`/`limit`. A cap therefore does not merely truncate one large read — it makes the -file unreachable *even through the offset/limit paging Pi's own tool description -advertises* ("Use offset/limit for large files… continue with offset until complete"). +file unreachable _even through the offset/limit paging Pi's own tool description +advertises_ ("Use offset/limit for large files… continue with offset until complete"). #180 describes the cost as "partially readable → unreadable"; it is actually "fully readable → unreadable by any means short of `bash`". §4.1 accepts that cost with an actionable error; §8 records the alternative. @@ -89,7 +89,12 @@ export interface ExecResult { export type ExecInPod = ( command: string, - opts?: { stdin?: Buffer; onData?: (chunk: Buffer) => void; signal?: AbortSignal; timeout?: number }, + opts?: { + stdin?: Buffer; + onData?: (chunk: Buffer) => void; + signal?: AbortSignal; + timeout?: number; + }, ) => Promise; ``` @@ -103,9 +108,9 @@ battery for every implementation. Retaining `exitCode: null` on truncation is what makes this **backward compatible**: every existing call site that checks `!== 0` keeps failing closed, so there is no flag day and no -call site *must* change. The flag adds precision on top. And the previously-ambiguous +call site _must_ change. The flag adds precision on top. And the previously-ambiguous combination stays meaningful in its remaining sense — `truncated: false` with -`exitCode: null` now means, unambiguously, "no exit status, and *not* because of our cap". +`exitCode: null` now means, unambiguously, "no exit status, and _not_ because of our cap". ### 3.2 `KubectlTransport`, `GrpcRelayTransport` @@ -120,7 +125,7 @@ for reasons specific to this transport: - The channel is **multiplexed**. `FrameParser` emits a frame only once both nonce markers have arrived, so "stop reading at the cap" would leave unparsed payload that corrupts the - *next* command's frames. The parser must reach the frame boundary regardless. + _next_ command's frames. The parser must reach the frame boundary regardless. - The payload is **base64**. Counting wire bytes caps content at cap × 3/4 ≈ 6 MiB, so the trip point would differ from the other two transports — a weaker version of the very distinguishability #180 objects to. @@ -134,14 +139,14 @@ Instead, extend `wrapCommand`'s pipeline so the pod caps its own output: This is better on four counts: -1. **Exact byte parity.** The cap applies to raw bytes *before* base64 inflation, so the +1. **Exact byte parity.** The cap applies to raw bytes _before_ base64 inflation, so the trip boundary is the same `> cap` as the other transports. 2. **`PIPESTATUS[0]` still indexes the command**, so the existing exit-code contract is untouched. A trip makes it 141 (SIGPIPE); we report `null` per the invariant and never read it, so truncation detection does not depend on 141 being distinguishable from a command's own internal SIGPIPE. 3. **It bounds a pre-existing O(n²).** `FrameParser.push` does `this.buf.toString("latin1")` - on *every* chunk, so a 20 MiB read allocates ~5.8 GB transiently. Capping the pod's + on _every_ chunk, so a 20 MiB read allocates ~5.8 GB transiently. Capping the pod's output bounds that buffer. A side benefit, not a goal — but it is why this transport should have been capped on performance grounds alone. 4. No cap-time channel teardown, so no respawn cost and no interaction with the retry path. @@ -152,7 +157,7 @@ and interpolates `cap + 1` into the pipeline. Without this the conformance batte exercise the cap at all — it injects `outputCapBytes: 6` (`conformance.ts:83`). Client-side, `persistentExecInPod` computes `truncated = frame.stdout.length > cap`, trims -to `cap`, appends `OUTPUT_TRUNCATED_MARKER`, and **resolves**. It must resolve, *not* route +to `cap`, appends `OUTPUT_TRUNCATED_MARKER`, and **resolves**. It must resolve, _not_ route through the existing `fail` handler: `fail` retries via `deps.fallback`, which `extension.ts:52` sets to the now-capped `KubectlTransport`, so a cap trip would re-run the command and flood twice before failing anyway. @@ -161,19 +166,19 @@ command and flood twice before failing anyway. GNU `coreutils` (verified: `head (GNU coreutils) 9.5`), and busybox `head` supports `-c` regardless. The framed protocol never runs against the remote worker — `run-leaf.ts:475` passes `transport: selected?.transport`, and `extension.ts:47-49` uses that single override -for *both* transports, so when a gRPC record is selected `persistentExecInPod` is not +for _both_ transports, so when a gRPC record is selected `persistentExecInPod` is not constructed at all. ### 3.4 Call sites -| Site | Change | -|---|---| -| `createPodBashOps` | `if (r.truncated) return { exitCode: 137 }` — **#181** | -| `readFile` | Split the overloaded null branch: `truncated` → size, cap, and the `bash`+`sed` range workaround; bare `null` → "signalled, no exit status" | -| `readFile` (size) | A truncated buffer cannot reveal the real file size, so on truncation only, `readFile` issues one `stat -c %s` to name it, omitting the size if that also fails. Error path only, so no hot-path cost — and the size is what lets the model pick a sensible range. | -| `glob` | Same split | -| `grep-tool.ts:44` | Name truncation instead of `rg failed in pod (exit null)` | -| `converge.ts:59`, `swebench-setup.ts:67` | Same, for `captureWorkspaceDiff` / `captureSwebenchDiff` — a >8 MiB diff currently reports `diff capture failed (exit null)` | +| Site | Change | +| ---------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | +| `createPodBashOps` | `if (r.truncated) return { exitCode: 137 }` — **#181** | +| `readFile` | Split the overloaded null branch: `truncated` → size, cap, and the `bash`+`sed` range workaround; bare `null` → "signalled, no exit status" | +| `readFile` (size) | A truncated buffer cannot reveal the real file size, so on truncation only, `readFile` issues one `stat -c %s` to name it, omitting the size if that also fails. Error path only, so no hot-path cost — and the size is what lets the model pick a sensible range. | +| `glob` | Same split | +| `grep-tool.ts:44` | Name truncation instead of `rg failed in pod (exit null)` | +| `converge.ts:59`, `swebench-setup.ts:67` | Same, for `captureWorkspaceDiff` / `captureSwebenchDiff` — a >8 MiB diff currently reports `diff capture failed (exit null)` | **Why 137 rather than a throw.** 137 is 128+9, the conventional SIGKILL status, and the command genuinely was killed by signal 9 at the cap — it is not a fabricated code. Pi's @@ -195,20 +200,20 @@ changes is only that the killing is now reported. `runConformance` gains a declared-capability object: ```ts -type ProducerStop = "remote-abort" | "local-kill" | "producer-side-cap" | "none"; +type ProducerStop = 'remote-abort' | 'local-kill' | 'producer-side-cap' | 'none'; -runConformance("KubectlTransport", make, { producerStop: "local-kill", streams: true }) -runConformance("GrpcRelayTransport", make, { producerStop: "remote-abort", streams: true }) -runConformance("persistentExecInPod", make, { producerStop: "producer-side-cap", streams: false }) +runConformance('KubectlTransport', make, { producerStop: 'local-kill', streams: true }); +runConformance('GrpcRelayTransport', make, { producerStop: 'remote-abort', streams: true }); +runConformance('persistentExecInPod', make, { producerStop: 'producer-side-cap', streams: false }); ``` - `producerStopped(): boolean` becomes `producerStop(): ProducerStop`. Each factory reports - the mechanism it *observed*, and the battery asserts it equals the declared value, instead + the mechanism it _observed_, and the battery asserts it equals the declared value, instead of accepting `true` from everyone. `"none"` is what a transport that stops nothing reports; no transport may declare it, so a regression that deletes the stop becomes a failure rather than a silent `true` — **#185**. - **Why an enum and not a "was it remote?" boolean.** A boolean would assert `false` for both - kubectl paths, which *discards* the existing coverage that `child.kill` is actually called — + kubectl paths, which _discards_ the existing coverage that `child.kill` is actually called — coverage #184's review found load-bearing (with it deleted, removing `child.kill` still passed). The enum keeps every transport pinned to a positive claim about its own mechanism. - `"producer-side-cap"` is the persistent channel's honest mechanism: pod-side `head -c` @@ -224,7 +229,7 @@ runConformance("persistentExecInPod", make, { producerStop: "producer-side-cap", **Honest note on the declaration.** Pod-side `head -c` bounds output at the source, and a producer that outruns it is stopped by SIGPIPE — the same EPIPE class #185 declines to count as "we stopped it". `producer-side-cap` therefore claims only what it delivers: the bytes -cannot exceed the cap, *not* that a hostile producer ignoring SIGPIPE stops burning CPU. It +cannot exceed the cap, _not_ that a hostile producer ignoring SIGPIPE stops burning CPU. It is deliberately not `remote-abort`, which is reserved for a transport that tells the far side to stop and can observe that it did. @@ -273,7 +278,7 @@ lands. - **Spec §8** — delete the "Known exception" paragraph in the Poisoned-output-defense bullet and both cap-related "Accepted divergences" entries (#181, #185); state the `truncated` contract, the invariant, and the per-transport producer-stop mechanism table. The #182 - default-deadline divergence stays. *Convention note:* the registry says specs are never + default-deadline divergence stays. _Convention note:_ the registry says specs are never retro-edited, but §8 is the live seam contract and #184 amended it on the same grounds; flagging it rather than assuming. - **ADR-0024** — a Revisions entry per decision, each with its rejected alternatives (§8). @@ -289,7 +294,7 @@ plus a test, currently CHANGES_REQUESTED for rejecting `rg`'s exit 1 ("no matche ## 8. Alternatives considered and rejected **#180 — cap the persistent channel at a higher, read-specific limit** (e.g. 64 MiB). -No realistic regression, still bounds memory. Rejected: a per-transport cap *value* leaves +No realistic regression, still bounds memory. Rejected: a per-transport cap _value_ leaves Pi able to distinguish backends by output volume between 8 and 64 MiB — a weaker form of the defect — and the battery would assert a parameter instead of a contract. @@ -302,7 +307,7 @@ the defect — and the battery would assert a parameter instead of a contract. the model's tool result (§3.4). **#181 — patch `pi-fork`'s `bash.ts` to recognize a truncation marker and append output.** -Best message *and* preserved output. Rejected: spans two repos, needs a fork-side commit and +Best message _and_ preserved output. Rejected: spans two repos, needs a fork-side commit and submodule rebuild, for a wording improvement over a mechanism (137) that already carries both facts. @@ -333,4 +338,4 @@ the property it exists to check, which is #185's actual complaint. --- -*Assisted-By: Claude (Anthropic AI) * +_Assisted-By: Claude (Anthropic AI) _ diff --git a/docs/specs/README.md b/docs/specs/README.md index 6f3e982..220e8a6 100644 --- a/docs/specs/README.md +++ b/docs/specs/README.md @@ -11,11 +11,11 @@ credential-plane re-examination. Two work streams accreted overlapping `M`-numbers: - The **built** harness work ran `M1–M7`. -- The **design-only** zero-trust credential plane (parent research doc) *also* started at `M7`, and +- The **design-only** zero-trust credential plane (parent research doc) _also_ started at `M7`, and later specs (`m10-mcp-code-mode`, `m13-…-egress`) reused/diverged from the parent's numbers. Net effect: `M7` meant two different things, `M10` meant two different things (the parent's MCP -*gateway* vs. the spec that *superseded* it with code-mode), `M13` was self-labeled "provisional," +_gateway_ vs. the spec that _superseded_ it with code-mode), `M13` was self-labeled "provisional," and the June-26 harness specs had no number at all. **Resolution:** freeze the built track as **Phase 1 (`M1–M7`)**; give the credential plane its own @@ -29,18 +29,18 @@ cross-references keep working. Pre-existing `M10`/`M13` labels are recorded here The decoupled scale-to-zero pattern. These are done and referenced across commits, memory, and `EXPERIMENTS.md`. **Frozen — do not renumber.** -| ID | Title | Spec | -|----|-------|------| -| M1 | Redis session backend | [`2026-06-16-m1-redis-session-backend-design.md`](2026-06-16-m1-redis-session-backend-design.md) | -| M2 | `K8sSandboxClient` (Pi Operations → remote pod) | [`2026-06-17-m2-k8s-sandbox-client-design.md`](2026-06-17-m2-k8s-sandbox-client-design.md) | -| M3 | Persistent in-pod channel | [`2026-06-17-m3-persistent-channel-design.md`](2026-06-17-m3-persistent-channel-design.md) | -| M4 | Knative serverless wrapper (`runTurn`) | [`2026-06-17-m4-knative-serverless-wrapper-design.md`](2026-06-17-m4-knative-serverless-wrapper-design.md) | -| M5 | Compaction-checkpoint fast path + budget voter | [`2026-06-23-m5-compaction-checkpoint-design.md`](2026-06-23-m5-compaction-checkpoint-design.md) | -| M6 | Experiments E2/E5 (`@sh/experiments`) | [`2026-06-24-m6-experiments-design.md`](2026-06-24-m6-experiments-design.md) | -| M7 | Cluster experiments E1/E3/E4 | [`2026-06-25-m7-cluster-experiments-design.md`](2026-06-25-m7-cluster-experiments-design.md) | +| ID | Title | Spec | +| --- | ----------------------------------------------- | ---------------------------------------------------------------------------------------------------------- | +| M1 | Redis session backend | [`2026-06-16-m1-redis-session-backend-design.md`](2026-06-16-m1-redis-session-backend-design.md) | +| M2 | `K8sSandboxClient` (Pi Operations → remote pod) | [`2026-06-17-m2-k8s-sandbox-client-design.md`](2026-06-17-m2-k8s-sandbox-client-design.md) | +| M3 | Persistent in-pod channel | [`2026-06-17-m3-persistent-channel-design.md`](2026-06-17-m3-persistent-channel-design.md) | +| M4 | Knative serverless wrapper (`runTurn`) | [`2026-06-17-m4-knative-serverless-wrapper-design.md`](2026-06-17-m4-knative-serverless-wrapper-design.md) | +| M5 | Compaction-checkpoint fast path + budget voter | [`2026-06-23-m5-compaction-checkpoint-design.md`](2026-06-23-m5-compaction-checkpoint-design.md) | +| M6 | Experiments E2/E5 (`@sh/experiments`) | [`2026-06-24-m6-experiments-design.md`](2026-06-24-m6-experiments-design.md) | +| M7 | Cluster experiments E1/E3/E4 | [`2026-06-25-m7-cluster-experiments-design.md`](2026-06-25-m7-cluster-experiments-design.md) | -> **Collision note:** Phase-1 `M7` (*cluster experiments*, built) is **not** the parent doc's `M7` -> (*egress/identity spine*, design). The latter is now **Z1** below. +> **Collision note:** Phase-1 `M7` (_cluster experiments_, built) is **not** the parent doc's `M7` +> (_egress/identity spine_, design). The latter is now **Z1** below. --- @@ -52,12 +52,12 @@ the [Capability Charter](2026-06-26-leaf-session-backend-capability-charter.md) promote-post-MVP (human-gate, cron trigger). They are the **MVP/charter track**, distinct from the `M`-numbered built harness (Phase 1) and the `Z`-numbered credential plane (Phase 2). -| Slice | Spec | PR(s) | -|----|-------|-------| +| Slice | Spec | PR(s) | +| -------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------- | -------- | | MVP leaf-session invocation contract (run-to-completion, structured output, volume envelope) + gate-7 durable resume | [`2026-06-26-mvp-leaf-session-contract-design.md`](2026-06-26-mvp-leaf-session-contract-design.md) | #10, #11 | -| Async leaf completion (KEDA `ScaledJob` + Redis Streams queue, done-marker) | [`2026-06-27-async-leaf-completion-design.md`](2026-06-27-async-leaf-completion-design.md) | #12 | -| Scheduled leaf dispatch (cron trigger on-ramp, Archetype C) | [`2026-06-28-scheduled-leaf-dispatch-design.md`](2026-06-28-scheduled-leaf-dispatch-design.md) | #13 | -| Human-gate (gate-while-idle, Archetype B) | [`2026-06-28-human-gate-design.md`](2026-06-28-human-gate-design.md) | #14 | +| Async leaf completion (KEDA `ScaledJob` + Redis Streams queue, done-marker) | [`2026-06-27-async-leaf-completion-design.md`](2026-06-27-async-leaf-completion-design.md) | #12 | +| Scheduled leaf dispatch (cron trigger on-ramp, Archetype C) | [`2026-06-28-scheduled-leaf-dispatch-design.md`](2026-06-28-scheduled-leaf-dispatch-design.md) | #13 | +| Human-gate (gate-while-idle, Archetype B) | [`2026-06-28-human-gate-design.md`](2026-06-28-human-gate-design.md) | #14 | All three archetypes (A parallel-fan-out, B human-gate, C scheduled) from the evidence base are now built. Hardening hygiene across these is tracked in @@ -70,18 +70,18 @@ built. Hardening hygiene across these is tracked in The [two-tier FS-free harness epic](https://github.com/kagenti/serverless-harness/issues/49): split the fleet into an FS-free **harness** (agent brain — credentials, model loop, network I/O only) and a durable **sandbox** (sole filesystem/syscall surface). Started as "run Archetype-A on OpenShift"; the -OCP RWX pain turned out to be a *symptom* of harness filesystem I/O, not the problem. Distinct from +OCP RWX pain turned out to be a _symptom_ of harness filesystem I/O, not the problem. Distinct from the `M`-numbered built harness (Phase 1) and the `Z`-numbered credential plane (Phase 2): this is an **architecture** track, dependency-ordered `P1 → P2 → P0′ → P3`. -| ID | Title | Status | Spec / issue | -|----|-------|--------|--------------| -| **P1** | **FS-free harness** — leaf envelope + human-gate off the filesystem (inline + Redis); sandbox working set `emptyDir` → agent-sandbox `Sandbox` CR durable PVC | **design ✅** | [`2026-07-02-p1-fs-free-harness-design.md`](2026-07-02-p1-fs-free-harness-design.md) (#45) | -| **P2** | **Shared sandbox pool + routing** — N distinct `Sandbox` CRs (per-sandbox RWO copy, no RWX), harness-side pick + Redis leases, ref-pinned lazy converge, static-N config knob | **design ✅** | [`2026-07-02-p2-shared-sandbox-pool-design.md`](2026-07-02-p2-shared-sandbox-pool-design.md) (#46) | -| **P0′** | OpenShift deployment of the FS-free harness — **P1 slice** (single durable RWO sandbox on OCP 4.20.8, full leaf smoke via Route) | **design ✅** | [`2026-07-02-p0prime-ocp-fs-free-deployment-design.md`](2026-07-02-p0prime-ocp-fs-free-deployment-design.md) (#47) | -| **P3** | **Sandbox sharing-ratio experiments** — measure the per-sandbox concurrency knee (→ `KAGENTI_SANDBOX_CAP`) and derived provisioning ratio N on runc; E6 saturation curve + E7 converge contention (delivers the deferred P2 live mixed-ref validation); in-cluster git-daemon substrate; Kind-dev → OCP-authoritative | **built ✅** (PR #58, OCP PR #61) | [`2026-07-03-p3-sandbox-sharing-ratio-experiments-design.md`](2026-07-03-p3-sandbox-sharing-ratio-experiments-design.md) (#48) | -| **P3.1** | **E6 workload-parameterized sandbox-load** — replace the trivial marker-check leaf with real Archetype-A code-review variants (L0/L1/L2); report N as a curve over per-leaf sandbox work (not one optimistic number); raise `max-scale`, warm baseline, multi-sample, sustained-decline `detectKnee` | **design ✅** | [`2026-07-03-e6-workload-parameterized-sandbox-load-design.md`](2026-07-03-e6-workload-parameterized-sandbox-load-design.md) (#62) | -| **P4** | Kata/VM isolation + intra-pod cross-leaf hardening (infra-gated: bare-metal pool vs Kata peer-pods vs gVisor — no nested KVM on the m6i cluster); Kata-overhead delta on P3's baseline; conditional RWX revisit | planned | #57 | +| ID | Title | Status | Spec / issue | +| -------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------- | +| **P1** | **FS-free harness** — leaf envelope + human-gate off the filesystem (inline + Redis); sandbox working set `emptyDir` → agent-sandbox `Sandbox` CR durable PVC | **design ✅** | [`2026-07-02-p1-fs-free-harness-design.md`](2026-07-02-p1-fs-free-harness-design.md) (#45) | +| **P2** | **Shared sandbox pool + routing** — N distinct `Sandbox` CRs (per-sandbox RWO copy, no RWX), harness-side pick + Redis leases, ref-pinned lazy converge, static-N config knob | **design ✅** | [`2026-07-02-p2-shared-sandbox-pool-design.md`](2026-07-02-p2-shared-sandbox-pool-design.md) (#46) | +| **P0′** | OpenShift deployment of the FS-free harness — **P1 slice** (single durable RWO sandbox on OCP 4.20.8, full leaf smoke via Route) | **design ✅** | [`2026-07-02-p0prime-ocp-fs-free-deployment-design.md`](2026-07-02-p0prime-ocp-fs-free-deployment-design.md) (#47) | +| **P3** | **Sandbox sharing-ratio experiments** — measure the per-sandbox concurrency knee (→ `KAGENTI_SANDBOX_CAP`) and derived provisioning ratio N on runc; E6 saturation curve + E7 converge contention (delivers the deferred P2 live mixed-ref validation); in-cluster git-daemon substrate; Kind-dev → OCP-authoritative | **built ✅** (PR #58, OCP PR #61) | [`2026-07-03-p3-sandbox-sharing-ratio-experiments-design.md`](2026-07-03-p3-sandbox-sharing-ratio-experiments-design.md) (#48) | +| **P3.1** | **E6 workload-parameterized sandbox-load** — replace the trivial marker-check leaf with real Archetype-A code-review variants (L0/L1/L2); report N as a curve over per-leaf sandbox work (not one optimistic number); raise `max-scale`, warm baseline, multi-sample, sustained-decline `detectKnee` | **design ✅** | [`2026-07-03-e6-workload-parameterized-sandbox-load-design.md`](2026-07-03-e6-workload-parameterized-sandbox-load-design.md) (#62) | +| **P4** | Kata/VM isolation + intra-pod cross-leaf hardening (infra-gated: bare-metal pool vs Kata peer-pods vs gVisor — no nested KVM on the m6i cluster); Kata-overhead delta on P3's baseline; conditional RWX revisit | planned | #57 | > **Supersedes** the local un-pushed `docs/archetype-a-ocp-support` branch (NFS-RWX-for-harness): > after P1 the harness mounts nothing, so the harness never co-mounts `/work`. Reference only. @@ -91,15 +91,15 @@ the `M`-numbered built harness (Phase 1) and the `Z`-numbered credential plane ( ## SandboxTransport (`ST`-prefix) Make the sandbox reachable from **anywhere** by inverting connectivity: the sandbox worker dials -*out* over one gRPC bidi stream (HTTP/2 on `:443`), a single-replica in-cluster relay bridges it to +_out_ over one gRPC bidi stream (HTTP/2 on `:443`), a single-replica in-cluster relay bridges it to the harness, and both paths land behind the existing `SandboxTransport` seam (`KubectlTransport` local + `GrpcRelayTransport` remote). Extends the shared-sandbox model (P2) to untrusted bring-your-own / NAT / on-prem / other-cloud sandboxes without touching the Pi loop, session backend, or leaf queue. Contract is a **language-neutral Protobuf IDL** (`sandbox/v1`), not a TypeScript interface. -| ID | Title | Status | Spec / issue | -|----|-------|--------|--------------| +| ID | Title | Status | Spec / issue | +| ------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **ST** | **SandboxTransport — language-neutral remote sandbox exec over gRPC** — worker-dialed `Attach` stream, single-replica presence-only relay mirroring into the existing pool, `SandboxTransport` seam, Go reference worker; per-sandbox bearer token day-one (SPIFFE/mTLS additive later) | **design ✅** | [`2026-07-08-sandbox-transport-grpc-design.md`](2026-07-08-sandbox-transport-grpc-design.md) (#78); [ADR-0024](../adrs/0024-sandbox-transport-remote-exec.md); epic #89 | > **Two build tracks, separate contributors.** The **backend** (TypeScript / in-cluster — proto, @@ -117,18 +117,18 @@ a TypeScript interface. > > **First buildable milestone:** [MVP Thin Slice — Leaf-Session Invocation Contract](2026-06-26-mvp-leaf-session-contract-design.md) (Archetype A) — proves an external orchestrator can dispatch N parallel, parameterized, run-to-completion leaf sessions with structured (volume-envelope) results, retry, and coverage audit, on scale-to-zero. Reuses M2–M6; defers the whole credential plane. -Principle (parent §2): *no component influenced by model output ever holds a raw secret.* Secrets +Principle (parent §2): _no component influenced by model output ever holds a raw secret._ Secrets live only in identity-keyed egress points. Dependency-ordered: -| ID | Title | Status | Spec / source | Alias | -|----|-------|--------|---------------|-------| -| **Z1** | Identity spine — per-session SPIFFE bound to user; `CredentialInjector` interface; orchestrator + reconstruct-on-wake | **design ✅** | [`2026-06-26-identity-spine-design.md`](2026-06-26-identity-spine-design.md) | parent M7 (reframed) | -| **Z2** | **Harness lock-down** — fail-closed redirection, secret-free container, default-deny egress, distroless, scoped RBAC; argues the harness needs **no** egress proxy | **design ✅** | [`2026-06-26-harness-lockdown-design.md`](2026-06-26-harness-lockdown-design.md) | — | -| **Z3** | **Inference injector** — shared provider-key chokepoint; multi-provider table, `x-sh-provider` routing, strip-then-set, mTLS, streaming, audit-only | **design ✅** · mechanism superseded by RC1 | [`2026-06-26-inference-injector-design.md`](2026-06-26-inference-injector-design.md) | parent M8 | -| **Z4** | MCP code-mode in the sandbox (placeholder-swap; **supersedes** the parent's MCP *gateway*) | design ✅ | [`2026-06-18-m10-mcp-code-mode-design.md`](2026-06-18-m10-mcp-code-mode-design.md) | M10 (spec); parent M10 (superseded) | -| **Z5** | Generalized credentialed egress (sandbox forward proxy + baked CA; subsumes the parent's sandbox-credential milestone; generalizes Z4's mechanism) | design ✅ · static slice implemented by RC1 | [`2026-06-19-m13-generalized-credentialed-egress-design.md`](2026-06-19-m13-generalized-credentialed-egress-design.md) | M13; parent M9 | -| **Z6** | Subagents — first-class child sessions; fresh-isolated default + `SandboxPolicy`; CoW workspace seed; `mail`/`subagent_*` log types | design (no spec yet) | parent research doc §3.4, §M11 | parent M11 | -| **Z7** | Validation — secret-leak red-team across all paths; multi-agent fan-out; blast-radius containment | design (no spec yet) | parent research doc §M12 | parent M12 | +| ID | Title | Status | Spec / source | Alias | +| ------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------- | ----------------------------------- | +| **Z1** | Identity spine — per-session SPIFFE bound to user; `CredentialInjector` interface; orchestrator + reconstruct-on-wake | **design ✅** | [`2026-06-26-identity-spine-design.md`](2026-06-26-identity-spine-design.md) | parent M7 (reframed) | +| **Z2** | **Harness lock-down** — fail-closed redirection, secret-free container, default-deny egress, distroless, scoped RBAC; argues the harness needs **no** egress proxy | **design ✅** | [`2026-06-26-harness-lockdown-design.md`](2026-06-26-harness-lockdown-design.md) | — | +| **Z3** | **Inference injector** — shared provider-key chokepoint; multi-provider table, `x-sh-provider` routing, strip-then-set, mTLS, streaming, audit-only | **design ✅** · mechanism superseded by RC1 | [`2026-06-26-inference-injector-design.md`](2026-06-26-inference-injector-design.md) | parent M8 | +| **Z4** | MCP code-mode in the sandbox (placeholder-swap; **supersedes** the parent's MCP _gateway_) | design ✅ | [`2026-06-18-m10-mcp-code-mode-design.md`](2026-06-18-m10-mcp-code-mode-design.md) | M10 (spec); parent M10 (superseded) | +| **Z5** | Generalized credentialed egress (sandbox forward proxy + baked CA; subsumes the parent's sandbox-credential milestone; generalizes Z4's mechanism) | design ✅ · static slice implemented by RC1 | [`2026-06-19-m13-generalized-credentialed-egress-design.md`](2026-06-19-m13-generalized-credentialed-egress-design.md) | M13; parent M9 | +| **Z6** | Subagents — first-class child sessions; fresh-isolated default + `SandboxPolicy`; CoW workspace seed; `mail`/`subagent_*` log types | design (no spec yet) | parent research doc §3.4, §M11 | parent M11 | +| **Z7** | Validation — secret-leak red-team across all paths; multi-agent fan-out; blast-radius containment | design (no spec yet) | parent research doc §M12 | parent M12 | ### Dependencies (Phase 2) @@ -153,8 +153,8 @@ profiles" pattern: a **shared** LLM gateway and a **per-sandbox** egress forward RFC 8693 deferred). Distinct from the `Z`-numbered plane it reframes — this is an **integration** track, so it takes its own prefix rather than a linear `Z` id. -| ID | Title | Status | Spec / decision | -|----|-------|--------|-----------------| +| ID | Title | Status | Spec / decision | +| ------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **RC1** | **AuthBridge egress control-plane PoC** — shared LLM gateway (Profile A) + per-sandbox egress forward-proxy (Profile B); real static-cred `token-broker` injection + stubbed-judge SPARC/IBAC control; BYO sandbox stretch on the `ST` seam (ST5-gated) | **accepted ✅** (2026-07-14) · RC1-0/1/2/4 implemented (Kind + OCP); RC1-3 stretch deferred (ST5) | [`2026-07-10-authbridge-egress-control-plane-poc-design.md`](2026-07-10-authbridge-egress-control-plane-poc-design.md); [ADR-0025](../adrs/0025-authbridge-deployment-topology.md) | --- @@ -172,7 +172,7 @@ so it takes its own prefix rather than a linear `Z` id. - The June-26 re-examination **reframed Z1's harness portion**: the harness gets a SPIFFE identity but **no egress waypoint** (its egress is fixed-destination; see Z2 §2.4). - **RC1** (own `RC` track) reframes the plane around **AuthBridge (Rosso Cortex)** as the mechanism, plus the `#89` - SandboxTransport seam. It **supersedes the *mechanism* of Z3** (the plain Go injector becomes an + SandboxTransport seam. It **supersedes the _mechanism_ of Z3** (the plain Go injector becomes an AuthBridge shared gateway once control plugins share the hop) and **implements a static single-tenant slice of Z5** (per-user / RFC 8693 token-exchange deferred), unifying both under one "egress control-plane, two deployment profiles" pattern. Z3/Z5 are retained as the deployment-profile detail @@ -182,14 +182,14 @@ so it takes its own prefix rather than a linear `Z` id. ## Documentation lifecycle -Three artifact types, three lifecycles. Code is the source of truth for *how*; the durable -value of docs is the *why* — decisions and the alternatives we rejected. +Three artifact types, three lifecycles. Code is the source of truth for _how_; the durable +value of docs is the _why_ — decisions and the alternatives we rejected. -| Artifact | Answers | Retention | Home | -|---|---|---|---| -| **Spec** (design doc) | *what & why* — alternatives, trade-offs, deferred work | committed, point-in-time; mark `Superseded by …`, don't delete | `docs/specs/` (here) | -| **ADR** | one significant decision + context + consequences | committed, **permanent & immutable**, supersession-aware | [`docs/adrs/`](../adrs/) | -| **Plan** (impl steps) | *how, in what order* | **local-only, ephemeral** — delete once coded; never committed | [`docs/plans/`](../plans/) (gitignored) | +| Artifact | Answers | Retention | Home | +| --------------------- | ------------------------------------------------------ | -------------------------------------------------------------- | --------------------------------------- | +| **Spec** (design doc) | _what & why_ — alternatives, trade-offs, deferred work | committed, point-in-time; mark `Superseded by …`, don't delete | `docs/specs/` (here) | +| **ADR** | one significant decision + context + consequences | committed, **permanent & immutable**, supersession-aware | [`docs/adrs/`](../adrs/) | +| **Plan** (impl steps) | _how, in what order_ | **local-only, ephemeral** — delete once coded; never committed | [`docs/plans/`](../plans/) (gitignored) | - A **spec** is a dated deep-dive. It's never retro-edited — a superseded spec gets a `Status:` header pointing at its successor and stays in git as the point-in-time record. @@ -221,4 +221,4 @@ gRPC/Connect sandbox-transport spec, which records the supersession in its own --- -*Assisted-By: Claude (Anthropic AI) * +_Assisted-By: Claude (Anthropic AI) _ diff --git a/experiments/README.md b/experiments/README.md index d460dd1..ecfe58f 100644 --- a/experiments/README.md +++ b/experiments/README.md @@ -28,12 +28,12 @@ leave the working tree dirty with machine-local timings after any `pnpm -r test` writes its own table to the gitignored `experiments/.results/` and compares only the columns that reproduce anywhere: -| Column | Compared? | Why | -|---|---|---| -| `N`, backend/checkpoint entries, ratio | yes | deterministic — identical on CI and a dev box | -| `backend bytes` | no | environment-sensitive (measured +4 bytes on CI) | -| `checkpoint bytes` | no | same class as `backend bytes` | -| `backend ms`, `checkpoint ms` | no | wall-clock; varies run to run | +| Column | Compared? | Why | +| -------------------------------------- | --------- | ----------------------------------------------- | +| `N`, backend/checkpoint entries, ratio | yes | deterministic — identical on CI and a dev box | +| `backend bytes` | no | environment-sensitive (measured +4 bytes on CI) | +| `checkpoint bytes` | no | same class as `backend bytes` | +| `backend ms`, `checkpoint ms` | no | wall-clock; varies run to run | So a change that moves the read counts fails E2 instead of silently rewriting the recorded result. When the move is legitimate, refresh the baseline deliberately and commit it: diff --git a/experiments/RESULTS.md b/experiments/RESULTS.md index af1a6d5..d20d8f6 100644 --- a/experiments/RESULTS.md +++ b/experiments/RESULTS.md @@ -15,11 +15,11 @@ run still matches those columns, and writes each run's own table to the gitignor counts: `SH_E2_UPDATE_BASELINE=1 pnpm -C experiments test e2-reconstruction-cost`. | N (session len) | backend entries | checkpoint entries | ratio (b/c) | backend bytes | checkpoint bytes | backend ms* | checkpoint ms* | -|---|---|---|---|---|---|---|---| -| 50 | 53 | 6 | 8.8 | 7482 | 896 | 0.9 | 1.2 | -| 200 | 203 | 6 | 33.8 | 28508 | 901 | 2.1 | 3.2 | -| 1000 | 1003 | 6 | 167.2 | 140908 | 901 | 4.8 | 5.1 | -| 5000 | 5003 | 6 | 833.8 | 706909 | 906 | 23.5 | 20.0 | +| --------------- | --------------- | ------------------ | ----------- | ------------- | ---------------- | ----------- | -------------- | +| 50 | 53 | 6 | 8.8 | 7482 | 896 | 0.9 | 1.2 | +| 200 | 203 | 6 | 33.8 | 28508 | 901 | 2.1 | 3.2 | +| 1000 | 1003 | 6 | 167.2 | 140908 | 901 | 4.8 | 5.1 | +| 5000 | 5003 | 6 | 833.8 | 706909 | 906 | 23.5 | 20.0 | **Pass:** checkpoint entries stay ~constant (bounded by the kept tail) while backend entries grow linearly with N, so the backend/checkpoint ratio strictly increases with N. `buildSessionContext()` @@ -33,4 +33,4 @@ with the cap unset the voter is inert (no block, no `abort`). A key-gated live r (`e5-budget-live.test.ts`, tiny cap) confirms the same end-to-end with a real model. See `README.md` for how to run the live variant. -*Assisted-By: Claude (Anthropic AI) * +_Assisted-By: Claude (Anthropic AI) _ diff --git a/experiments/src/counting-backend.ts b/experiments/src/counting-backend.ts index 57d6e8c..3e35796 100644 --- a/experiments/src/counting-backend.ts +++ b/experiments/src/counting-backend.ts @@ -1,4 +1,4 @@ -import type { FileEntry, SessionStorageBackend } from "@earendil-works/pi-coding-agent"; +import type { FileEntry, SessionStorageBackend } from '@earendil-works/pi-coding-agent'; export interface ReadCounts { reads: number; diff --git a/experiments/src/report.ts b/experiments/src/report.ts index a894a55..d905206 100644 --- a/experiments/src/report.ts +++ b/experiments/src/report.ts @@ -43,10 +43,10 @@ export function deterministicView(rows: E2Row[]): E2Deterministic[] { */ export function parseE2Table(markdown: string): E2Row[] { const rows: E2Row[] = []; - for (const line of markdown.split("\n")) { + for (const line of markdown.split('\n')) { const trimmed = line.trim(); - if (!trimmed.startsWith("|")) continue; - const cells = trimmed.slice(1, trimmed.endsWith("|") ? -1 : undefined).split("|"); + if (!trimmed.startsWith('|')) continue; + const cells = trimmed.slice(1, trimmed.endsWith('|') ? -1 : undefined).split('|'); if (cells.length !== 8) continue; // header, separator, and other tables const nums = cells.map((c) => Number(c.trim())); if (nums.some((v) => !Number.isFinite(v))) continue; // header/separator row @@ -72,21 +72,21 @@ export function parseE2Table(markdown: string): E2Row[] { }); } if (rows.length === 0) { - throw new Error("no E2 table found: expected the 8-column table buildResultsMarkdown emits"); + throw new Error('no E2 table found: expected the 8-column table buildResultsMarkdown emits'); } return rows; } export function buildResultsMarkdown(rows: E2Row[]): string { const header = - "| N (session len) | backend entries | checkpoint entries | ratio (b/c) | backend bytes | checkpoint bytes | backend ms* | checkpoint ms* |\n" + - "|---|---|---|---|---|---|---|---|"; + '| N (session len) | backend entries | checkpoint entries | ratio (b/c) | backend bytes | checkpoint bytes | backend ms* | checkpoint ms* |\n' + + '|---|---|---|---|---|---|---|---|'; const body = rows .map( (r) => `| ${r.n} | ${r.backendEntries} | ${r.checkpointEntries} | ${r.ratioEntries.toFixed(1)} | ${r.backendBytes} | ${r.checkpointBytes} | ${r.backendMs.toFixed(1)} | ${r.checkpointMs.toFixed(1)} |`, ) - .join("\n"); + .join('\n'); return `# M6 Experiment Results ## E2 — local reconstruction cost (openFromCheckpoint vs openFromBackend) diff --git a/experiments/src/session-fixture.ts b/experiments/src/session-fixture.ts index ed504b7..d8dcd29 100644 --- a/experiments/src/session-fixture.ts +++ b/experiments/src/session-fixture.ts @@ -1,6 +1,6 @@ -import { SessionManager, type FileEntry } from "@earendil-works/pi-coding-agent"; -import { RedisSessionBackend } from "@sh/session-backend"; -import { BufferedRedisBackend } from "@sh/harness/buffered-redis-backend"; +import { SessionManager, type FileEntry } from '@earendil-works/pi-coding-agent'; +import { RedisSessionBackend } from '@sh/session-backend'; +import { BufferedRedisBackend } from '@sh/harness/buffered-redis-backend'; export interface CompactedFixture { sessionId: string; @@ -27,21 +27,21 @@ export async function buildCompactedSession( const ids: string[] = []; for (let i = 0; i < opts.n; i++) { - const role = i % 2 === 0 ? "user" : "assistant"; + const role = i % 2 === 0 ? 'user' : 'assistant'; const id = sm.appendMessage({ role, content: `m${i}` } as never); ids.push(id); } // Keep the last `tailKept` messages: firstKept = the message at index n - tailKept. const firstKeptId = ids[Math.max(0, opts.n - tailKept)]; - sm.appendCompaction("summary of earlier turns", firstKeptId, 1234); + sm.appendCompaction('summary of earlier turns', firstKeptId, 1234); await backend.flush(); const resumeFromPosition = await store.positionOfId(sessionId, firstKeptId); if (resumeFromPosition == null) { throw new Error(`fixture: positionOfId returned null for firstKeptId ${firstKeptId}`); } - sm.appendCustomEntry("checkpoint", { resumeFromPosition }); + sm.appendCustomEntry('checkpoint', { resumeFromPosition }); await backend.flush(); const full = await store.read(sessionId); diff --git a/experiments/src/sharing.ts b/experiments/src/sharing.ts index 68d087e..d91f37b 100644 --- a/experiments/src/sharing.ts +++ b/experiments/src/sharing.ts @@ -1,7 +1,7 @@ export interface LadderPoint { - c: number; // concurrent leaves at this rung + c: number; // concurrent leaves at this rung throughput: number; // aggregate leaves/sec - p95Ms: number; // per-leaf p95 latency at this rung + p95Ms: number; // per-leaf p95 latency at this rung } /** @@ -12,7 +12,7 @@ export interface LadderPoint { */ export function detectKnee(points: LadderPoint[], degradeX: number, patience = 2): number { const baseline = points.find((p) => p.c === 1); - if (!baseline) throw new Error("detectKnee: no c=1 baseline point"); + if (!baseline) throw new Error('detectKnee: no c=1 baseline point'); const bound = baseline.p95Ms * degradeX; const sorted = [...points].sort((a, b) => a.c - b.c); let knee = 1; @@ -34,13 +34,13 @@ export function detectKnee(points: LadderPoint[], degradeX: number, patience = 2 /** Fraction of wall-clock the sandbox was busy on this leaf's execs; in (0, 1]. */ export function dutyCycle(execBusyMs: number, wallMs: number): number { - if (wallMs <= 0) throw new Error("dutyCycle: wallMs must be > 0"); + if (wallMs <= 0) throw new Error('dutyCycle: wallMs must be > 0'); return Math.min(1, execBusyMs / wallMs); } /** How many such leaves time-share one sandbox before it is continuously busy. */ export function derivedRatio(duty: number): number { - if (duty <= 0) throw new Error("derivedRatio: duty must be > 0"); + if (duty <= 0) throw new Error('derivedRatio: duty must be > 0'); return Math.round((1 / duty) * 10) / 10; } @@ -51,7 +51,7 @@ export function sanityFloorPass(knee: number, minConcurrency: number): boolean { export interface LeafObservation { runId: string; - expectedRef: string; // the ref this leaf's envelope pinned + expectedRef: string; // the ref this leaf's envelope pinned observedMarker: string; // marker.txt content read from its worktree } @@ -66,9 +66,9 @@ export function worktreeConsistent(obs: LeafObservation[]): { export interface WorkloadPoint { label: string; - execMs: number; // sandbox-busy ms attributable to one leaf of this workload + execMs: number; // sandbox-busy ms attributable to one leaf of this workload execCount: number; // number of sandbox execs the leaf issued - wallMs: number; // leaf wall-clock + wallMs: number; // leaf wall-clock } export interface RatioCurvePoint { @@ -87,7 +87,7 @@ export function buildRatioCurve(points: WorkloadPoint[]): RatioCurvePoint[] { } export interface ArmResult { - arm: "dedicated" | "shared"; + arm: 'dedicated' | 'shared'; resvSecPerLeaf: number; p95Ms: number; throughput: number; @@ -100,7 +100,8 @@ export function reservationBenefit( shared: ArmResult, degradeX: number, ): { ratio: number; withinDegrade: boolean } { - if (shared.resvSecPerLeaf <= 0) throw new Error("reservationBenefit: shared resvSecPerLeaf must be > 0"); + if (shared.resvSecPerLeaf <= 0) + throw new Error('reservationBenefit: shared resvSecPerLeaf must be > 0'); const ratio = Math.round((dedicated.resvSecPerLeaf / shared.resvSecPerLeaf) * 10) / 10; return { ratio, withinDegrade: shared.p95Ms <= dedicated.p95Ms * degradeX }; } diff --git a/experiments/src/workload.ts b/experiments/src/workload.ts index caf8b86..16971d1 100644 --- a/experiments/src/workload.ts +++ b/experiments/src/workload.ts @@ -1,30 +1,47 @@ -import { readFileSync } from "node:fs"; +import { readFileSync } from 'node:fs'; export interface DeckInstance { - instance_id: string; repo: string; base_commit: string; env_key: string; - problem_statement: string; weight_bucket: "light" | "medium" | "heavy" | null; + instance_id: string; + repo: string; + base_commit: string; + env_key: string; + problem_statement: string; + weight_bucket: 'light' | 'medium' | 'heavy' | null; test_runtime_ms: number | null; } -export interface Deck { deckHash: string; seed: number; instances: DeckInstance[] } +export interface Deck { + deckHash: string; + seed: number; + instances: DeckInstance[]; +} -export interface WorkItem { instanceId: string; label: string; post: Record } -export interface SliceOpts { perBucket: number; buckets?: string[]; seed?: number } +export interface WorkItem { + instanceId: string; + label: string; + post: Record; +} +export interface SliceOpts { + perBucket: number; + buckets?: string[]; + seed?: number; +} export interface WorkloadProvider { - readonly name: "synthetic" | "swebench"; + readonly name: 'synthetic' | 'swebench'; curveItems(): WorkItem[]; sweepItem(): WorkItem; sliceItems(opts: SliceOpts): WorkItem[]; } export function loadDeck(path: string): Deck { - return JSON.parse(readFileSync(path, "utf8")) as Deck; + return JSON.parse(readFileSync(path, 'utf8')) as Deck; } // Deterministic PRNG (mulberry32) so a (seed) reproduces the same selection without a dependency. function rng(seed: number): () => number { let a = seed >>> 0; return () => { - a |= 0; a = (a + 0x6d2b79f5) | 0; + a |= 0; + a = (a + 0x6d2b79f5) | 0; let t = Math.imul(a ^ (a >>> 15), 1 | a); t = (t + Math.imul(t ^ (t >>> 7), 61 | t)) ^ t; return ((t ^ (t >>> 14)) >>> 0) / 4294967296; @@ -32,28 +49,40 @@ function rng(seed: number): () => number { } function shuffle(arr: T[], rand: () => number): T[] { const a = [...arr]; - for (let i = a.length - 1; i > 0; i--) { const j = Math.floor(rand() * (i + 1)); [a[i], a[j]] = [a[j], a[i]]; } + for (let i = a.length - 1; i > 0; i--) { + const j = Math.floor(rand() * (i + 1)); + [a[i], a[j]] = [a[j], a[i]]; + } return a; } -const BUCKETS = ["light", "medium", "heavy"] as const; +const BUCKETS = ['light', 'medium', 'heavy'] as const; class SyntheticProvider implements WorkloadProvider { - readonly name = "synthetic" as const; + readonly name = 'synthetic' as const; private readonly variants = [ - { label: "L0", file: "small.py", pattern: "password" }, - { label: "L1", file: "medium.py", pattern: "eval(" }, - { label: "L2", file: "large.py", pattern: "eval(" }, + { label: 'L0', file: 'small.py', pattern: 'password' }, + { label: 'L1', file: 'medium.py', pattern: 'eval(' }, + { label: 'L2', file: 'large.py', pattern: 'eval(' }, ]; private toItem(v: { label: string; file: string; pattern: string }): WorkItem { - return { instanceId: v.label, label: v.label, - post: { item: { item_id: v.label, file: v.file, pattern: v.pattern } } }; + return { + instanceId: v.label, + label: v.label, + post: { item: { item_id: v.label, file: v.file, pattern: v.pattern } }, + }; + } + curveItems(): WorkItem[] { + return this.variants.map((v) => this.toItem(v)); + } + sweepItem(): WorkItem { + return this.toItem(this.variants[this.variants.length - 1]); } - curveItems(): WorkItem[] { return this.variants.map((v) => this.toItem(v)); } - sweepItem(): WorkItem { return this.toItem(this.variants[this.variants.length - 1]); } sliceItems(opts: SliceOpts): WorkItem[] { // Synthetic has one item per "bucket" (variant); replicate perBucket times for a slice run. - return this.variants.flatMap((v) => Array.from({ length: opts.perBucket }, () => this.toItem(v))); + return this.variants.flatMap((v) => + Array.from({ length: opts.perBucket }, () => this.toItem(v)), + ); } } @@ -61,10 +90,14 @@ class SyntheticProvider implements WorkloadProvider { // fails on the baked swebench image — C-ext / build-heavy repos (verified live: matplotlib fails // setup in ~19s; sklearn/numpy build likewise). A curve representative must actually solve setup, so // representative() skips these. sweepItem() already avoids matplotlib by ranking on test_runtime_ms. -const SETUP_UNRELIABLE_REPOS = new Set(["matplotlib/matplotlib", "scikit-learn/scikit-learn", "numpy/numpy"]); +const SETUP_UNRELIABLE_REPOS = new Set([ + 'matplotlib/matplotlib', + 'scikit-learn/scikit-learn', + 'numpy/numpy', +]); class SwebenchProvider implements WorkloadProvider { - readonly name = "swebench" as const; + readonly name = 'swebench' as const; constructor(private readonly deck: Deck) {} private byBucket(bucket: string): DeckInstance[] { // Sort by instance_id for a stable base order independent of deck ordering. @@ -75,9 +108,9 @@ class SwebenchProvider implements WorkloadProvider { private toItem(inst: DeckInstance): WorkItem { return { instanceId: inst.instance_id, - label: inst.weight_bucket ?? "unknown", + label: inst.weight_bucket ?? 'unknown', post: { - kind: "solve", + kind: 'solve', problemStatement: inst.problem_statement, repoUrl: `/repos/${inst.repo}.git`, ref: inst.base_commit, @@ -101,8 +134,8 @@ class SwebenchProvider implements WorkloadProvider { sweepItem(): WorkItem { // The heaviest single instance = largest test_runtime_ms in the heavy bucket (null treated as -1), // tie-broken by instance_id ascending for determinism. - this.representative("heavy"); // throws a helpful error if the heavy bucket is empty - const heavy = this.byBucket("heavy"); + this.representative('heavy'); // throws a helpful error if the heavy bucket is empty + const heavy = this.byBucket('heavy'); const best = heavy.reduce((a, b) => { const ar = a.test_runtime_ms ?? -1; const br = b.test_runtime_ms ?? -1; @@ -115,7 +148,11 @@ class SwebenchProvider implements WorkloadProvider { sliceItems(opts: SliceOpts): WorkItem[] { const buckets = opts.buckets ?? [...BUCKETS]; const rand = rng(opts.seed ?? this.deck.seed ?? 1); - return buckets.flatMap((b) => shuffle(this.byBucket(b), rand).slice(0, opts.perBucket).map((i) => this.toItem(i))); + return buckets.flatMap((b) => + shuffle(this.byBucket(b), rand) + .slice(0, opts.perBucket) + .map((i) => this.toItem(i)), + ); } } @@ -128,9 +165,9 @@ export function getWorkloadProvider( env: Record, deckPath?: string, ): WorkloadProvider { - const which = (env.WORKLOAD ?? "synthetic").toLowerCase(); - if (which === "swebench") { - const path = deckPath ?? env.DECK ?? "experiments/swebench/deck.json"; + const which = (env.WORKLOAD ?? 'synthetic').toLowerCase(); + if (which === 'swebench') { + const path = deckPath ?? env.DECK ?? 'experiments/swebench/deck.json'; return new SwebenchProvider(loadDeck(path)); } return new SyntheticProvider(); diff --git a/experiments/swebench/RUNBOOK.md b/experiments/swebench/RUNBOOK.md index 74b7635..60008e3 100644 --- a/experiments/swebench/RUNBOOK.md +++ b/experiments/swebench/RUNBOOK.md @@ -11,21 +11,22 @@ contributors who have never run it before. Two experiments over [SWE-bench Verified](https://www.swebench.com/) solve tasks ("leaves"), each of which leases an execution **sandbox** (a pod with the repo + tools) from a shared pool: -| Exp | Question | Output | -|-----|----------|--------| -| **E6** | How many leaves can one sandbox serve (sharing ratio **N = 1/duty**)? Where's the concurrency knee? | `RATIO_CURVE`, `E6_RESULT`, sweep points | -| **E1** | Dedicated (1 sandbox/leaf) vs a shared pool: reservation-seconds saved? | `E1B_RESULT` (benefit ratio) | -| **Plan D** | Are the produced patches *correct*? (offline, model-free) | resolved-rate | +| Exp | Question | Output | +| ---------- | --------------------------------------------------------------------------------------------------- | ---------------------------------------- | +| **E6** | How many leaves can one sandbox serve (sharing ratio **N = 1/duty**)? Where's the concurrency knee? | `RATIO_CURVE`, `E6_RESULT`, sweep points | +| **E1** | Dedicated (1 sandbox/leaf) vs a shared pool: reservation-seconds saved? | `E1B_RESULT` (benefit ratio) | +| **Plan D** | Are the produced patches _correct_? (offline, model-free) | resolved-rate | **Platform constraint:** the SWE-bench sandbox image is **x86_64-only** (built from the official -per-instance eval images), so the *cluster* steps are **OpenShift/AMD64 only**. The *offline -evaluator* (§9) runs anywhere with Docker, including arm64 laptops under emulation. +per-instance eval images), so the _cluster_ steps are **OpenShift/AMD64 only**. The _offline +evaluator_ (§9) runs anywhere with Docker, including arm64 laptops under emulation. --- ## 1. Prerequisites **Cluster** + - OpenShift **4.20+**, AMD64 worker nodes, `oc` logged in as **cluster-admin**. - Reference run used 6× `m6i.xlarge` (4 vCPU / 16 GiB) on AWS. Sandbox pods request 512Mi/250m, limit 4Gi; each sandbox needs a **50Gi RWO PVC** (a default StorageClass must exist). @@ -35,6 +36,7 @@ evaluator* (§9) runs anywhere with Docker, including arm64 laptops under emulat (for §9 only). **Credentials**: an Anthropic API key. Export **before** cluster setup: + ```bash export ANTHROPIC_API_KEY=sk-ant-... # direct Anthropic (recommended) # — or, for a gateway: export ANTHROPIC_AUTH_TOKEN=... ANTHROPIC_BASE_URL=https://... @@ -49,6 +51,7 @@ git clone https://github.com/kagenti/serverless-harness.git cd serverless-harness git submodule update --init --recursive # pi-fork submodule ``` + All paths below are relative to this repo root. Requires the SWE-bench experiment scripts on `main` (Plan B/C + the Plan D evaluator, PR #140). @@ -75,6 +78,7 @@ oc patch knativeserving knative-serving -n knative-serving --type merge \ ``` Sanity check: + ```bash oc get ksvc serverless-harness -n default # Ready=True, URL present oc get pods -n default -l app=redis # Running @@ -82,6 +86,7 @@ oc get crd sandboxes.agents.x-k8s.io # installed ``` Capture the route for the drivers: + ```bash export KSVC_URL="https://$(oc get route serverless-harness -n default -o jsonpath='{.spec.host}')" export KUBECONFIG= @@ -121,6 +126,7 @@ deploy/knative/build-swebench-sandbox.sh --emit --limit 5 --offset 5 \ ``` Confirm the final `…ff962cb83fe5c624-15of15` tag exists: + ```bash oc get istag -n default | grep swebench-sandbox ``` @@ -154,6 +160,7 @@ oc apply -f deploy/knative/swebench-sandbox-pool.yaml oc get sandbox -n default -l app=sandbox # wait for swebench-sandbox-0/1/2 oc wait --for=condition=Ready pod -l sh.kagenti.io/sandbox-pool=swebench -n default --timeout=600s ``` + 3 Sandbox CRs, pool label `sh.kagenti.io/sandbox-pool=swebench`, image pinned to the `…-15of15` internal-registry tag, 50Gi PVC each. First image pull takes several minutes. @@ -162,6 +169,7 @@ oc wait --for=condition=Ready pod -l sh.kagenti.io/sandbox-pool=swebench -n defa ## 7. Run the experiments Set a per-run log dir and keep everything under it: + ```bash export LOG_DIR=/tmp/kagenti/run-$(date +%Y%m%d); mkdir -p "$LOG_DIR" ``` @@ -176,6 +184,7 @@ E6_LIVE=1 WORKLOAD=swebench \ KSVC_URL="$KSVC_URL" KUBECONFIG="$KUBECONFIG" LOG_DIR="$LOG_DIR" \ bash deploy/knative/e6-saturation.sh 2>&1 | tee "$LOG_DIR/e6.log" ``` + Emits per-bucket duty, `RATIO_CURVE=…`, `sweep c=… p95Ms=…`, a final `E6_RESULT …`, and a `COST_REPORT …`. Note the `knee=` value. @@ -192,6 +201,7 @@ E1B_LIVE=1 \ KSVC_URL="$KSVC_URL" KUBECONFIG="$KUBECONFIG" LOG_DIR="$LOG_DIR" \ bash deploy/knative/e1-benefit.sh 2>&1 | tee "$LOG_DIR/e1.log" ``` + Emits `dedicated:` / `shared@N:` lines, a final `E1B_RESULT benefit=…x …`, and a `COST_REPORT …`. > **Long runs (~4–6 h):** run each driver under `nohup … **Health tally** on `E6_RESULT`/`E1B_RESULT` (`health=solved/total … transport=N`): OpenShift -> ingress may drop long-lived HTTP responses; those leaves are *excluded from metrics but counted*. +> ingress may drop long-lived HTTP responses; those leaves are _excluded from metrics but counted_. > Their cost is still captured (Redis-side) and their patch is recoverable (`poll_leaf_result`), so > the numbers stay trustworthy despite transport loss. @@ -239,12 +249,14 @@ PRED_A="$LOG_DIR/predictions.jsonl" PRED_B="$LOG_DIR/predictions-e1.jsonl" \ RUN_ID=my-run MAX_WORKERS=2 LOG_DIR="$LOG_DIR/eval" \ bash experiments/swebench/evaluate.sh ``` + Prints `RESOLVED_RATE = / (…%)` plus the resolved / unresolved / empty-patch / errored instance lists. The script installs `swebench` in a venv, merges + dedups the two prediction files (restoring trailing newlines), pulls prebuilt x86 eval images (`--namespace swebench`), and summarizes the report. Smoke one instance first (fast, proves the loop): + ```bash INSTANCE_IDS="django__django-11555" RUN_ID=smoke MAX_WORKERS=1 \ PRED_A="$LOG_DIR/predictions.jsonl" PRED_B="$LOG_DIR/predictions-e1.jsonl" \ @@ -276,13 +288,13 @@ oc get sandbox -n default -l app=sandbox # default pool back to 3/3 ## 11. Interpreting the results -| Line | Meaning | -|------|---------| -| `RATIO_CURVE … n=5.7/7.3/3.1` | **N = 1/duty** per weight bucket = concurrent leaves one sandbox could serve. Reference: light 5.7 / medium 7.3 / heavy 3.1. | -| `sweep c=… p95Ms=…` | Concurrency sweep. If p95 climbs while throughput is flat and sandboxes stay lightly loaded → the **knee is upstream** (model/harness tier), not the sandbox pool. | -| `E1B_RESULT benefit=1.5x` | Dedicated ÷ shared reservation-seconds/leaf. >1 = pooling saves reserved sandbox-time; check `withinDegrade=true` (p95 within the 2× budget). | -| `COST_REPORT costUsd=…` | Model spend, summed from Redis session streams (dominated by cache-reads). Reference full run ≈ $25.90 on Haiku. | -| `RESOLVED_RATE` (§9) | Correctness: fraction of applied patches that pass the gold tests. | +| Line | Meaning | +| ----------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------ | +| `RATIO_CURVE … n=5.7/7.3/3.1` | **N = 1/duty** per weight bucket = concurrent leaves one sandbox could serve. Reference: light 5.7 / medium 7.3 / heavy 3.1. | +| `sweep c=… p95Ms=…` | Concurrency sweep. If p95 climbs while throughput is flat and sandboxes stay lightly loaded → the **knee is upstream** (model/harness tier), not the sandbox pool. | +| `E1B_RESULT benefit=1.5x` | Dedicated ÷ shared reservation-seconds/leaf. >1 = pooling saves reserved sandbox-time; check `withinDegrade=true` (p95 within the 2× budget). | +| `COST_REPORT costUsd=…` | Model spend, summed from Redis session streams (dominated by cache-reads). Reference full run ≈ $25.90 on Haiku. | +| `RESOLVED_RATE` (§9) | Correctness: fraction of applied patches that pass the gold tests. | **Sizing guidance from the reference run:** set the shared-pool cap from the **duty-cycle N (3–7)**, and scale **harness pods + model quota** (not the sandbox pool) for more throughput. @@ -291,15 +303,15 @@ and scale **harness pods + model quota** (not the sandbox pool) for more through ## 12. Troubleshooting -| Symptom | Cause / fix | -|---------|-------------| -| `set_ksvc_timeout` fails / revision not Ready | Cluster `max-revision-timeout-seconds` < 1800 — do the §3 patch. | -| Leaves killed mid-run, empty patches | Same timeout issue, or ingress dropped the response. Check the `transport=` tally; cost/patch are still recoverable from Redis. | -| `SandboxPoolSaturatedError` | Every pool pod at `KAGENTI_SANDBOX_CAP`. Raise the cap or add Sandbox CRs. | -| Evaluator: `patch unexpectedly ends in middle of line` | Missing trailing newline on captured patch. `merge_predictions.py` repairs it; the capture-side fix is PR #141. | -| Evaluator very slow / flaky on arm64 | x86 images under emulation. Lower `MAX_WORKERS`, or run on a native x86 host. Pure-python instances work; C-extension repos need native x86. | -| Sandbox pod `ImagePullBackOff` | The `…-15of15` istag isn't built/pushed (§4), or the pool YAML tag doesn't match. | +| Symptom | Cause / fix | +| ------------------------------------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------- | +| `set_ksvc_timeout` fails / revision not Ready | Cluster `max-revision-timeout-seconds` < 1800 — do the §3 patch. | +| Leaves killed mid-run, empty patches | Same timeout issue, or ingress dropped the response. Check the `transport=` tally; cost/patch are still recoverable from Redis. | +| `SandboxPoolSaturatedError` | Every pool pod at `KAGENTI_SANDBOX_CAP`. Raise the cap or add Sandbox CRs. | +| Evaluator: `patch unexpectedly ends in middle of line` | Missing trailing newline on captured patch. `merge_predictions.py` repairs it; the capture-side fix is PR #141. | +| Evaluator very slow / flaky on arm64 | x86 images under emulation. Lower `MAX_WORKERS`, or run on a native x86 host. Pure-python instances work; C-extension repos need native x86. | +| Sandbox pod `ImagePullBackOff` | The `…-15of15` istag isn't built/pushed (§4), or the pool YAML tag doesn't match. | --- -*Reference run: 2026-07-17, OpenShift 4.20.8, Claude Haiku 4.5. Assisted-By: Claude Code.* +_Reference run: 2026-07-17, OpenShift 4.20.8, Claude Haiku 4.5. Assisted-By: Claude Code._ diff --git a/experiments/swebench/deck.json b/experiments/swebench/deck.json index 8853f1d..1075ebf 100644 --- a/experiments/swebench/deck.json +++ b/experiments/swebench/deck.json @@ -43,10 +43,7 @@ "test_stop_start_slicing (ordering.tests.OrderingTests)" ], "test_cmd": "./tests/runtests.py --verbosity 2 --settings=test_sqlite --parallel 1", - "test_directives": [ - "ordering.models", - "ordering.tests" - ], + "test_directives": ["ordering.models", "ordering.tests"], "test_runtime_ms": 945, "weight_bucket": "medium" }, @@ -81,9 +78,7 @@ "test_uuid_unsupported (model_enums.tests.CustomChoicesTests)" ], "test_cmd": "./tests/runtests.py --verbosity 2 --settings=test_sqlite --parallel 1", - "test_directives": [ - "model_enums.tests" - ], + "test_directives": ["model_enums.tests"], "test_runtime_ms": 715, "weight_bucket": "light" }, @@ -101,9 +96,7 @@ ], "pass_to_pass": [], "test_cmd": "./tests/runtests.py --verbosity 2 --settings=test_sqlite --parallel 1", - "test_directives": [ - "project_template.test_settings" - ], + "test_directives": ["project_template.test_settings"], "test_runtime_ms": 905, "weight_bucket": "medium" }, @@ -163,9 +156,7 @@ "test_webp (files.tests.GetImageDimensionsTests)" ], "test_cmd": "./tests/runtests.py --verbosity 2 --settings=test_sqlite --parallel 1", - "test_directives": [ - "files.tests" - ], + "test_directives": ["files.tests"], "test_runtime_ms": 685, "weight_bucket": "light" }, @@ -214,9 +205,7 @@ "test_process_view_return_response (middleware_exceptions.tests.AsyncMiddlewareTests)" ], "test_cmd": "./tests/runtests.py --verbosity 2 --settings=test_sqlite --parallel 1", - "test_directives": [ - "middleware_exceptions.tests" - ], + "test_directives": ["middleware_exceptions.tests"], "test_runtime_ms": 736, "weight_bucket": "light" }, @@ -517,10 +506,7 @@ "test_tickets_7448_7707 (queries.tests.Queries1Tests)" ], "test_cmd": "./tests/runtests.py --verbosity 2 --settings=test_sqlite --parallel 1", - "test_directives": [ - "queries.models", - "queries.tests" - ], + "test_directives": ["queries.models", "queries.tests"], "test_runtime_ms": 1720, "weight_bucket": "medium" }, @@ -533,9 +519,7 @@ "env_key": "sweb.env.py.x86_64.934a137824256b612e9dc5:latest", "problem_statement": "filter on exists-subquery with empty queryset removes whole WHERE block\nDescription\n\t \n\t\t(last modified by Tobias Bengfort)\n\t \n>>> qs = MyModel.objects.filter(~models.Exists(MyModel.objects.none()), name='test')\n>>> qs\n\n>>> print(qs.query)\nEmptyResultSet\nWith django-debug-toolbar I can still see the query, but there WHERE block is missing completely.\nThis seems to be very similar to #33018.\n", "test_patch": "diff --git a/tests/expressions/tests.py b/tests/expressions/tests.py\n--- a/tests/expressions/tests.py\n+++ b/tests/expressions/tests.py\n@@ -1905,6 +1905,13 @@ def test_optimizations(self):\n )\n self.assertNotIn('ORDER BY', captured_sql)\n \n+ def test_negated_empty_exists(self):\n+ manager = Manager.objects.create()\n+ qs = Manager.objects.filter(\n+ ~Exists(Manager.objects.none()) & Q(pk=manager.pk)\n+ )\n+ self.assertSequenceEqual(qs, [manager])\n+\n \n class FieldTransformTests(TestCase):\n \n", - "fail_to_pass": [ - "test_negated_empty_exists (expressions.tests.ExistsTests)" - ], + "fail_to_pass": ["test_negated_empty_exists (expressions.tests.ExistsTests)"], "pass_to_pass": [ "test_equal (expressions.tests.OrderByTests)", "test_hash (expressions.tests.OrderByTests)", @@ -700,9 +684,7 @@ "test_uuid_pk_subquery (expressions.tests.BasicExpressionsTests)" ], "test_cmd": "./tests/runtests.py --verbosity 2 --settings=test_sqlite --parallel 1", - "test_directives": [ - "expressions.tests" - ], + "test_directives": ["expressions.tests"], "test_runtime_ms": 1061, "weight_bucket": "medium" }, @@ -799,9 +781,7 @@ "test_values_with_pk_annotation (annotations.tests.NonAggregateAnnotationTestCase)" ], "test_cmd": "./tests/runtests.py --verbosity 2 --settings=test_sqlite --parallel 1", - "test_directives": [ - "annotations.tests" - ], + "test_directives": ["annotations.tests"], "test_runtime_ms": 977, "weight_bucket": "medium" }, @@ -856,9 +836,7 @@ "`obj` is passed from `InlineModelAdmin.get_fieldsets()` to" ], "test_cmd": "./tests/runtests.py --verbosity 2 --settings=test_sqlite --parallel 1", - "test_directives": [ - "modeladmin.tests" - ], + "test_directives": ["modeladmin.tests"], "test_runtime_ms": 868, "weight_bucket": "light" }, @@ -871,9 +849,7 @@ "env_key": "sweb.env.py.x86_64.31244378a92e3bcce809ac:latest", "problem_statement": "[Bug]: ax.hist density not auto-scaled when using histtype='step'\n### Bug summary\r\n\r\nI need to plot a histogram of some data (generated by `numpy.save` in binary format) from my work using the `matplotlib.axes.Axes.hist` function. I noted that the histogram's density axis (when setting `density=True`) is not automatically adjusted to fit the whole histogram. \r\n\r\nI played with different combinations of parameters, and noted that the densities changes if you rescale the whole data array, which is counterintuitive as rescaling the data should only affect the x-axis values. I noted that if you set `histtype=\"step\"`, the issue will occur, but is otherwise okay for other `histtype`s.\r\n\r\nI started a github repo for testing this issue [here](https://github.com/coryzh/matplotlib_3.6_hist_bug_report). The `test.npy `file is the data generated from my program.\r\n\r\n### Code for reproduction\r\n\r\n```python\r\nscale = 1.2\r\ntest_random = np.random.randn(100000) * scale\r\n\r\nfig, ax = plt.subplots(1, 2, figsize=(20, 10))\r\nhist_bar = ax[0].hist(test_random, bins=100, density=True, histtype=\"bar\")\r\nhist_step = ax[1].hist(test_random, bins=100, density=True, histtype=\"step\")\r\nplt.show()\r\n```\r\n\r\n\r\n### Actual outcome\r\n\r\nHere's the histograms generated using some simulated data. You can play with the `histtype` and `scale` parameters in the code to see the differences. When `scale=1.2`, I got\r\n![histogram_test_actual](https://user-images.githubusercontent.com/32777663/194084553-2ee3a8dc-c78b-4827-b292-d2bee828076f.png)\r\n\r\n\r\n### Expected outcome\r\nWhen `scale=1`, sometimes the randomised array would lead to identical left and right panel ...\r\n![histogram_test_expected](https://user-images.githubusercontent.com/32777663/194084586-3748f64e-97fc-4f32-b0f1-9526e8e8dcec.png)\r\n\r\n\r\n### Additional information\r\n\r\n\r\n_No response_\r\n\r\n### Operating system\r\n\r\nOS/X\r\n\r\n### Matplotlib Version\r\n\r\n3.6.0\r\n\r\n### Matplotlib Backend\r\n\r\n_No response_\r\n\r\n### Python version\r\n\r\n3.10.4\r\n\r\n### Jupyter version\r\n\r\n_No response_\r\n\r\n### Installation\r\n\r\npip\n", "test_patch": "diff --git a/lib/matplotlib/tests/test_axes.py b/lib/matplotlib/tests/test_axes.py\n--- a/lib/matplotlib/tests/test_axes.py\n+++ b/lib/matplotlib/tests/test_axes.py\n@@ -8165,6 +8165,58 @@ def test_bezier_autoscale():\n assert ax.get_ylim()[0] == -0.5\n \n \n+def test_small_autoscale():\n+ # Check that paths with small values autoscale correctly #24097.\n+ verts = np.array([\n+ [-5.45, 0.00], [-5.45, 0.00], [-5.29, 0.00], [-5.29, 0.00],\n+ [-5.13, 0.00], [-5.13, 0.00], [-4.97, 0.00], [-4.97, 0.00],\n+ [-4.81, 0.00], [-4.81, 0.00], [-4.65, 0.00], [-4.65, 0.00],\n+ [-4.49, 0.00], [-4.49, 0.00], [-4.33, 0.00], [-4.33, 0.00],\n+ [-4.17, 0.00], [-4.17, 0.00], [-4.01, 0.00], [-4.01, 0.00],\n+ [-3.85, 0.00], [-3.85, 0.00], [-3.69, 0.00], [-3.69, 0.00],\n+ [-3.53, 0.00], [-3.53, 0.00], [-3.37, 0.00], [-3.37, 0.00],\n+ [-3.21, 0.00], [-3.21, 0.01], [-3.05, 0.01], [-3.05, 0.01],\n+ [-2.89, 0.01], [-2.89, 0.01], [-2.73, 0.01], [-2.73, 0.02],\n+ [-2.57, 0.02], [-2.57, 0.04], [-2.41, 0.04], [-2.41, 0.04],\n+ [-2.25, 0.04], [-2.25, 0.06], [-2.09, 0.06], [-2.09, 0.08],\n+ [-1.93, 0.08], [-1.93, 0.10], [-1.77, 0.10], [-1.77, 0.12],\n+ [-1.61, 0.12], [-1.61, 0.14], [-1.45, 0.14], [-1.45, 0.17],\n+ [-1.30, 0.17], [-1.30, 0.19], [-1.14, 0.19], [-1.14, 0.22],\n+ [-0.98, 0.22], [-0.98, 0.25], [-0.82, 0.25], [-0.82, 0.27],\n+ [-0.66, 0.27], [-0.66, 0.29], [-0.50, 0.29], [-0.50, 0.30],\n+ [-0.34, 0.30], [-0.34, 0.32], [-0.18, 0.32], [-0.18, 0.33],\n+ [-0.02, 0.33], [-0.02, 0.32], [0.13, 0.32], [0.13, 0.33], [0.29, 0.33],\n+ [0.29, 0.31], [0.45, 0.31], [0.45, 0.30], [0.61, 0.30], [0.61, 0.28],\n+ [0.77, 0.28], [0.77, 0.25], [0.93, 0.25], [0.93, 0.22], [1.09, 0.22],\n+ [1.09, 0.19], [1.25, 0.19], [1.25, 0.17], [1.41, 0.17], [1.41, 0.15],\n+ [1.57, 0.15], [1.57, 0.12], [1.73, 0.12], [1.73, 0.10], [1.89, 0.10],\n+ [1.89, 0.08], [2.05, 0.08], [2.05, 0.07], [2.21, 0.07], [2.21, 0.05],\n+ [2.37, 0.05], [2.37, 0.04], [2.53, 0.04], [2.53, 0.02], [2.69, 0.02],\n+ [2.69, 0.02], [2.85, 0.02], [2.85, 0.01], [3.01, 0.01], [3.01, 0.01],\n+ [3.17, 0.01], [3.17, 0.00], [3.33, 0.00], [3.33, 0.00], [3.49, 0.00],\n+ [3.49, 0.00], [3.65, 0.00], [3.65, 0.00], [3.81, 0.00], [3.81, 0.00],\n+ [3.97, 0.00], [3.97, 0.00], [4.13, 0.00], [4.13, 0.00], [4.29, 0.00],\n+ [4.29, 0.00], [4.45, 0.00], [4.45, 0.00], [4.61, 0.00], [4.61, 0.00],\n+ [4.77, 0.00], [4.77, 0.00], [4.93, 0.00], [4.93, 0.00],\n+ ])\n+\n+ minx = np.min(verts[:, 0])\n+ miny = np.min(verts[:, 1])\n+ maxx = np.max(verts[:, 0])\n+ maxy = np.max(verts[:, 1])\n+\n+ p = mpath.Path(verts)\n+\n+ fig, ax = plt.subplots()\n+ ax.add_patch(mpatches.PathPatch(p))\n+ ax.autoscale()\n+\n+ assert ax.get_xlim()[0] <= minx\n+ assert ax.get_xlim()[1] >= maxx\n+ assert ax.get_ylim()[0] <= miny\n+ assert ax.get_ylim()[1] >= maxy\n+\n+\n def test_get_xticklabel():\n fig, ax = plt.subplots()\n ax.plot(np.arange(10))\n", - "fail_to_pass": [ - "lib/matplotlib/tests/test_axes.py::test_small_autoscale" - ], + "fail_to_pass": ["lib/matplotlib/tests/test_axes.py::test_small_autoscale"], "pass_to_pass": [ "lib/matplotlib/tests/test_axes.py::test_invisible_axes[png]", "lib/matplotlib/tests/test_axes.py::test_get_labels", @@ -1645,9 +1621,7 @@ "lib/matplotlib/tests/test_axes.py::test_bar_all_nan[png]" ], "test_cmd": "pytest -rA", - "test_directives": [ - "lib/matplotlib/tests/test_axes.py" - ], + "test_directives": ["lib/matplotlib/tests/test_axes.py"], "test_runtime_ms": 2277, "weight_bucket": "heavy" }, @@ -2030,9 +2004,7 @@ "xarray/tests/test_variable.py::TestBackendIndexing::test_DaskIndexingAdapter" ], "test_cmd": "pytest -rA", - "test_directives": [ - "xarray/tests/test_variable.py" - ], + "test_directives": ["xarray/tests/test_variable.py"], "test_runtime_ms": 26746, "weight_bucket": "heavy" }, @@ -2045,9 +2017,7 @@ "env_key": "sweb.env.py.x86_64.6b6c43248aa28ed62e8334:latest", "problem_statement": "`xr.where(..., keep_attrs=True)` overwrites coordinate attributes\n### What happened?\n\n#6461 had some unintended consequences for `xr.where(..., keep_attrs=True)`, where coordinate attributes are getting overwritten by variable attributes. I guess this has been broken since `2022.06.0`.\n\n### What did you expect to happen?\n\nCoordinate attributes should be preserved.\n\n### Minimal Complete Verifiable Example\n\n```Python\nimport xarray as xr\r\nds = xr.tutorial.load_dataset(\"air_temperature\")\r\nxr.where(True, ds.air, ds.air, keep_attrs=True).time.attrs\n```\n\n\n### MVCE confirmation\n\n- [X] Minimal example \u2014 the example is as focused as reasonably possible to demonstrate the underlying issue in xarray.\n- [X] Complete example \u2014 the example is self-contained, including all data and the text of any traceback.\n- [X] Verifiable example \u2014 the example copy & pastes into an IPython prompt or [Binder notebook](https://mybinder.org/v2/gh/pydata/xarray/main?urlpath=lab/tree/doc/examples/blank_template.ipynb), returning the result.\n- [X] New issue \u2014 a search of GitHub Issues suggests this is not a duplicate.\n\n### Relevant log output\n\n```Python\n# New time attributes are:\r\n{'long_name': '4xDaily Air temperature at sigma level 995',\r\n 'units': 'degK',\r\n 'precision': 2,\r\n 'GRIB_id': 11,\r\n 'GRIB_name': 'TMP',\r\n 'var_desc': 'Air temperature',\r\n 'dataset': 'NMC Reanalysis',\r\n 'level_desc': 'Surface',\r\n 'statistic': 'Individual Obs',\r\n 'parent_stat': 'Other',\r\n 'actual_range': array([185.16, 322.1 ], dtype=float32)}\r\n\r\n# Instead of:\r\n{'standard_name': 'time', 'long_name': 'Time'}\n```\n\n\n### Anything else we need to know?\n\nI'm struggling to figure out how the simple `lambda` change in #6461 brought this about. I tried tracing my way through the various merge functions but there are a lot of layers. Happy to submit a PR if someone has an idea for an obvious fix.\n\n### Environment\n\n

\r\nINSTALLED VERSIONS\r\n------------------\r\ncommit: None\r\npython: 3.9.13 | packaged by conda-forge | (main, May 27 2022, 16:56:21) \r\n[GCC 10.3.0]\r\npython-bits: 64\r\nOS: Linux\r\nOS-release: 5.15.0-52-generic\r\nmachine: x86_64\r\nprocessor: x86_64\r\nbyteorder: little\r\nLC_ALL: None\r\nLANG: en_US.UTF-8\r\nLOCALE: ('en_US', 'UTF-8')\r\nlibhdf5: 1.12.2\r\nlibnetcdf: 4.8.1\r\n\r\nxarray: 2022.10.0\r\npandas: 1.4.3\r\nnumpy: 1.23.4\r\nscipy: 1.9.3\r\nnetCDF4: 1.6.1\r\npydap: None\r\nh5netcdf: 1.0.2\r\nh5py: 3.7.0\r\nNio: None\r\nzarr: 2.13.3\r\ncftime: 1.6.2\r\nnc_time_axis: 1.4.1\r\nPseudoNetCDF: None\r\nrasterio: 1.3.3\r\ncfgrib: 0.9.10.2\r\niris: None\r\nbottleneck: 1.3.5\r\ndask: 2022.10.0\r\ndistributed: 2022.10.0\r\nmatplotlib: 3.6.1\r\ncartopy: 0.21.0\r\nseaborn: None\r\nnumbagg: None\r\nfsspec: 2022.10.0\r\ncupy: None\r\npint: 0.19.2\r\nsparse: 0.13.0\r\nflox: 0.6.1\r\nnumpy_groupies: 0.9.19\r\nsetuptools: 65.5.0\r\npip: 22.3\r\nconda: None\r\npytest: 7.1.3\r\nIPython: 8.5.0\r\nsphinx: None\r\n\r\n\r\n\r\n
\r\n\n", "test_patch": "diff --git a/xarray/tests/test_computation.py b/xarray/tests/test_computation.py\n--- a/xarray/tests/test_computation.py\n+++ b/xarray/tests/test_computation.py\n@@ -1925,16 +1925,63 @@ def test_where() -> None:\n \n \n def test_where_attrs() -> None:\n- cond = xr.DataArray([True, False], dims=\"x\", attrs={\"attr\": \"cond\"})\n- x = xr.DataArray([1, 1], dims=\"x\", attrs={\"attr\": \"x\"})\n- y = xr.DataArray([0, 0], dims=\"x\", attrs={\"attr\": \"y\"})\n+ cond = xr.DataArray([True, False], coords={\"a\": [0, 1]}, attrs={\"attr\": \"cond_da\"})\n+ cond[\"a\"].attrs = {\"attr\": \"cond_coord\"}\n+ x = xr.DataArray([1, 1], coords={\"a\": [0, 1]}, attrs={\"attr\": \"x_da\"})\n+ x[\"a\"].attrs = {\"attr\": \"x_coord\"}\n+ y = xr.DataArray([0, 0], coords={\"a\": [0, 1]}, attrs={\"attr\": \"y_da\"})\n+ y[\"a\"].attrs = {\"attr\": \"y_coord\"}\n+\n+ # 3 DataArrays, takes attrs from x\n actual = xr.where(cond, x, y, keep_attrs=True)\n- expected = xr.DataArray([1, 0], dims=\"x\", attrs={\"attr\": \"x\"})\n+ expected = xr.DataArray([1, 0], coords={\"a\": [0, 1]}, attrs={\"attr\": \"x_da\"})\n+ expected[\"a\"].attrs = {\"attr\": \"x_coord\"}\n assert_identical(expected, actual)\n \n- # ensure keep_attrs can handle scalar values\n+ # x as a scalar, takes no attrs\n+ actual = xr.where(cond, 0, y, keep_attrs=True)\n+ expected = xr.DataArray([0, 0], coords={\"a\": [0, 1]})\n+ assert_identical(expected, actual)\n+\n+ # y as a scalar, takes attrs from x\n+ actual = xr.where(cond, x, 0, keep_attrs=True)\n+ expected = xr.DataArray([1, 0], coords={\"a\": [0, 1]}, attrs={\"attr\": \"x_da\"})\n+ expected[\"a\"].attrs = {\"attr\": \"x_coord\"}\n+ assert_identical(expected, actual)\n+\n+ # x and y as a scalar, takes no attrs\n actual = xr.where(cond, 1, 0, keep_attrs=True)\n- assert actual.attrs == {}\n+ expected = xr.DataArray([1, 0], coords={\"a\": [0, 1]})\n+ assert_identical(expected, actual)\n+\n+ # cond and y as a scalar, takes attrs from x\n+ actual = xr.where(True, x, y, keep_attrs=True)\n+ expected = xr.DataArray([1, 1], coords={\"a\": [0, 1]}, attrs={\"attr\": \"x_da\"})\n+ expected[\"a\"].attrs = {\"attr\": \"x_coord\"}\n+ assert_identical(expected, actual)\n+\n+ # DataArray and 2 Datasets, takes attrs from x\n+ ds_x = xr.Dataset(data_vars={\"x\": x}, attrs={\"attr\": \"x_ds\"})\n+ ds_y = xr.Dataset(data_vars={\"x\": y}, attrs={\"attr\": \"y_ds\"})\n+ ds_actual = xr.where(cond, ds_x, ds_y, keep_attrs=True)\n+ ds_expected = xr.Dataset(\n+ data_vars={\n+ \"x\": xr.DataArray([1, 0], coords={\"a\": [0, 1]}, attrs={\"attr\": \"x_da\"})\n+ },\n+ attrs={\"attr\": \"x_ds\"},\n+ )\n+ ds_expected[\"a\"].attrs = {\"attr\": \"x_coord\"}\n+ assert_identical(ds_expected, ds_actual)\n+\n+ # 2 DataArrays and 1 Dataset, takes attrs from x\n+ ds_actual = xr.where(cond, x.rename(\"x\"), ds_y, keep_attrs=True)\n+ ds_expected = xr.Dataset(\n+ data_vars={\n+ \"x\": xr.DataArray([1, 0], coords={\"a\": [0, 1]}, attrs={\"attr\": \"x_da\"})\n+ },\n+ )\n+ ds_expected[\"a\"].attrs = {\"attr\": \"x_coord\"}\n+ assert_identical(ds_expected, ds_actual)\n \n \n @pytest.mark.parametrize(\n", - "fail_to_pass": [ - "xarray/tests/test_computation.py::test_where_attrs" - ], + "fail_to_pass": ["xarray/tests/test_computation.py::test_where_attrs"], "pass_to_pass": [ "xarray/tests/test_computation.py::test_signature_properties", "xarray/tests/test_computation.py::test_result_name", @@ -2331,9 +2301,7 @@ "xarray/tests/test_computation.py::test_cross[a6-b6-ae6-be6-cartesian--1-True]" ], "test_cmd": "pytest -rA", - "test_directives": [ - "xarray/tests/test_computation.py" - ], + "test_directives": ["xarray/tests/test_computation.py"], "test_runtime_ms": 16822, "weight_bucket": "heavy" }, @@ -2346,9 +2314,7 @@ "env_key": "sweb.env.py.x86_64.b70be1b28e254d81bab6a9:latest", "problem_statement": "`pylint` removes first item from `sys.path` when running from `runpy`.\n### Bug description\n\nThis is the line where the first item from sys.path is removed.\r\nhttps://github.com/PyCQA/pylint/blob/ce7cccf96454fb6e286e4a8f38919733a0f28f44/pylint/__init__.py#L99\r\n\r\nI think there should be a check to ensure that the first item is `\"\"`, `\".\"` or `os.getcwd()` before removing.\n\n### Configuration\n\n_No response_\n\n### Command used\n\n```shell\nRun programmatically to repro this, using this code:\r\n\r\nimport sys\r\nimport runpy\r\n\r\nsys.path.insert(0, \"something\")\r\n\r\nrunpy.run_module('pylint', run_name=\"__main__\", alter_sys=True)\n```\n\n\n### Pylint output\n\n```shell\nWhen using pylint extension which bundles the libraries, the extension add them to sys.path depending on user settings. Pylint removes the first entry from sys path causing it to fail to load.\n```\n\n\n### Expected behavior\n\nCheck if `\"\"`, `\".\"` or `os.getcwd()` before removing the first item from sys.path\n\n### Pylint version\n\n```shell\npylint 2.14.5\n```\n\n\n### OS / Environment\n\n_No response_\n\n### Additional dependencies\n\n_No response_\n", "test_patch": "diff --git a/tests/test_self.py b/tests/test_self.py\n--- a/tests/test_self.py\n+++ b/tests/test_self.py\n@@ -759,6 +759,24 @@ def test_modify_sys_path() -> None:\n modify_sys_path()\n assert sys.path == paths[1:]\n \n+ paths = [\"\", *default_paths]\n+ sys.path = copy(paths)\n+ with _test_environ_pythonpath():\n+ modify_sys_path()\n+ assert sys.path == paths[1:]\n+\n+ paths = [\".\", *default_paths]\n+ sys.path = copy(paths)\n+ with _test_environ_pythonpath():\n+ modify_sys_path()\n+ assert sys.path == paths[1:]\n+\n+ paths = [\"/do_not_remove\", *default_paths]\n+ sys.path = copy(paths)\n+ with _test_environ_pythonpath():\n+ modify_sys_path()\n+ assert sys.path == paths\n+\n paths = [cwd, cwd, *default_paths]\n sys.path = copy(paths)\n with _test_environ_pythonpath(\".\"):\n", - "fail_to_pass": [ - "tests/test_self.py::TestRunTC::test_modify_sys_path" - ], + "fail_to_pass": ["tests/test_self.py::TestRunTC::test_modify_sys_path"], "pass_to_pass": [ "tests/test_self.py::TestRunTC::test_pkginfo", "tests/test_self.py::TestRunTC::test_all", @@ -2474,9 +2440,7 @@ "tests/test_self.py::TestCallbackOptions::test_enable_all_extensions" ], "test_cmd": "pytest -rA", - "test_directives": [ - "tests/test_self.py" - ], + "test_directives": ["tests/test_self.py"], "test_runtime_ms": 431, "weight_bucket": "light" }, @@ -2489,9 +2453,7 @@ "env_key": "sweb.env.py.x86_64.7f83a0ba1392a745c061ce:latest", "problem_statement": "caplog.get_records and caplog.clear conflict\n# Description\r\n\r\n`caplog.get_records()` gets decoupled from actual caplog records when `caplog.clear()` is called. As a result, after `caplog.clear()` is called, `caplog.get_records()` is frozen: it does not get cleared, nor does it get new records.\r\n\r\nDuring test set up it is [set to the same list](https://github.com/pytest-dev/pytest/blob/28e8c8582ea947704655a3c3f2d57184831336fd/src/_pytest/logging.py#L699) as `caplog.records`, but the latter gets [replaced rather than cleared](https://github.com/pytest-dev/pytest/blob/28e8c8582ea947704655a3c3f2d57184831336fd/src/_pytest/logging.py#L345) in `caplog.clear()`, which diverges the two objects.\r\n\r\n# Reproductive example\r\n```python\r\nimport logging\r\n\r\ndef test(caplog) -> None:\r\n def verify_consistency() -> None:\r\n assert caplog.get_records(\"call\") == caplog.records\r\n\r\n verify_consistency()\r\n logging.warning(\"test\")\r\n verify_consistency()\r\n caplog.clear()\r\n verify_consistency() # fails: assert [] == []\r\n```\r\n\r\n# Environment details\r\nArch Linux, Python 3.9.10:\r\n```\r\nPackage Version\r\n---------- -------\r\nattrs 21.4.0\r\niniconfig 1.1.1\r\npackaging 21.3\r\npip 22.0.4\r\npluggy 1.0.0\r\npy 1.11.0\r\npyparsing 3.0.8\r\npytest 7.1.1\r\nsetuptools 60.10.0\r\ntomli 2.0.1\r\nwheel 0.37.1\r\n```\n", "test_patch": "diff --git a/testing/logging/test_fixture.py b/testing/logging/test_fixture.py\n--- a/testing/logging/test_fixture.py\n+++ b/testing/logging/test_fixture.py\n@@ -172,6 +172,24 @@ def test_caplog_captures_for_all_stages(caplog, logging_during_setup_and_teardow\n assert set(caplog._item.stash[caplog_records_key]) == {\"setup\", \"call\"}\n \n \n+def test_clear_for_call_stage(caplog, logging_during_setup_and_teardown):\n+ logger.info(\"a_call_log\")\n+ assert [x.message for x in caplog.get_records(\"call\")] == [\"a_call_log\"]\n+ assert [x.message for x in caplog.get_records(\"setup\")] == [\"a_setup_log\"]\n+ assert set(caplog._item.stash[caplog_records_key]) == {\"setup\", \"call\"}\n+\n+ caplog.clear()\n+\n+ assert caplog.get_records(\"call\") == []\n+ assert [x.message for x in caplog.get_records(\"setup\")] == [\"a_setup_log\"]\n+ assert set(caplog._item.stash[caplog_records_key]) == {\"setup\", \"call\"}\n+\n+ logging.info(\"a_call_log_after_clear\")\n+ assert [x.message for x in caplog.get_records(\"call\")] == [\"a_call_log_after_clear\"]\n+ assert [x.message for x in caplog.get_records(\"setup\")] == [\"a_setup_log\"]\n+ assert set(caplog._item.stash[caplog_records_key]) == {\"setup\", \"call\"}\n+\n+\n def test_ini_controls_global_log_level(pytester: Pytester) -> None:\n pytester.makepyfile(\n \"\"\"\n", - "fail_to_pass": [ - "testing/logging/test_fixture.py::test_clear_for_call_stage" - ], + "fail_to_pass": ["testing/logging/test_fixture.py::test_clear_for_call_stage"], "pass_to_pass": [ "testing/logging/test_fixture.py::test_change_level", "testing/logging/test_fixture.py::test_with_statement", @@ -2510,9 +2472,7 @@ "testing/logging/test_fixture.py::test_log_report_captures_according_to_config_option_upon_failure" ], "test_cmd": "pytest -rA", - "test_directives": [ - "testing/logging/test_fixture.py" - ], + "test_directives": ["testing/logging/test_fixture.py"], "test_runtime_ms": 557, "weight_bucket": "light" }, @@ -2525,9 +2485,7 @@ "env_key": "sweb.env.py.x86_64.aa92880033da20ca313928:latest", "problem_statement": "IndexError: list index out of range in export_text when the tree only has one feature\n\r\n\r\n\r\n\r\n#### Description\r\n`export_text` returns `IndexError` when there is single feature.\r\n\r\n#### Steps/Code to Reproduce\r\n```python\r\nfrom sklearn.tree import DecisionTreeClassifier\r\nfrom sklearn.tree.export import export_text\r\nfrom sklearn.datasets import load_iris\r\n\r\nX, y = load_iris(return_X_y=True)\r\nX = X[:, 0].reshape(-1, 1)\r\n\r\ntree = DecisionTreeClassifier()\r\ntree.fit(X, y)\r\ntree_text = export_text(tree, feature_names=['sepal_length'])\r\nprint(tree_text)\r\n\r\n```\r\n\r\n#### Actual Results\r\n```\r\nIndexError: list index out of range\r\n```\r\n\r\n\r\n#### Versions\r\n```\r\nCould not locate executable g77\r\nCould not locate executable f77\r\nCould not locate executable ifort\r\nCould not locate executable ifl\r\nCould not locate executable f90\r\nCould not locate executable DF\r\nCould not locate executable efl\r\nCould not locate executable gfortran\r\nCould not locate executable f95\r\nCould not locate executable g95\r\nCould not locate executable efort\r\nCould not locate executable efc\r\nCould not locate executable flang\r\ndon't know how to compile Fortran code on platform 'nt'\r\n\r\nSystem:\r\n python: 3.7.3 (default, Apr 24 2019, 15:29:51) [MSC v.1915 64 bit (AMD64)]\r\nexecutable: C:\\Users\\liqia\\Anaconda3\\python.exe\r\n machine: Windows-10-10.0.17763-SP0\r\n\r\nBLAS:\r\n macros: \r\n lib_dirs: \r\ncblas_libs: cblas\r\n\r\nPython deps:\r\n pip: 19.1\r\nsetuptools: 41.0.0\r\n sklearn: 0.21.1\r\n numpy: 1.16.2\r\n scipy: 1.2.1\r\n Cython: 0.29.7\r\n pandas: 0.24.2\r\nC:\\Users\\liqia\\Anaconda3\\lib\\site-packages\\numpy\\distutils\\system_info.py:638: UserWarning: \r\n Atlas (http://math-atlas.sourceforge.net/) libraries not found.\r\n Directories to search for the libraries can be specified in the\r\n numpy/distutils/site.cfg file (section [atlas]) or by setting\r\n the ATLAS environment variable.\r\n self.calc_info()\r\nC:\\Users\\liqia\\Anaconda3\\lib\\site-packages\\numpy\\distutils\\system_info.py:638: UserWarning: \r\n Blas (http://www.netlib.org/blas/) libraries not found.\r\n Directories to search for the libraries can be specified in the\r\n numpy/distutils/site.cfg file (section [blas]) or by setting\r\n the BLAS environment variable.\r\n self.calc_info()\r\nC:\\Users\\liqia\\Anaconda3\\lib\\site-packages\\numpy\\distutils\\system_info.py:638: UserWarning: \r\n Blas (http://www.netlib.org/blas/) sources not found.\r\n Directories to search for the sources can be specified in the\r\n numpy/distutils/site.cfg file (section [blas_src]) or by setting\r\n the BLAS_SRC environment variable.\r\n self.calc_info()\r\n```\r\n\r\n\r\n\n", "test_patch": "diff --git a/sklearn/tree/tests/test_export.py b/sklearn/tree/tests/test_export.py\n--- a/sklearn/tree/tests/test_export.py\n+++ b/sklearn/tree/tests/test_export.py\n@@ -396,6 +396,21 @@ def test_export_text():\n assert export_text(reg, decimals=1) == expected_report\n assert export_text(reg, decimals=1, show_weights=True) == expected_report\n \n+ X_single = [[-2], [-1], [-1], [1], [1], [2]]\n+ reg = DecisionTreeRegressor(max_depth=2, random_state=0)\n+ reg.fit(X_single, y_mo)\n+\n+ expected_report = dedent(\"\"\"\n+ |--- first <= 0.0\n+ | |--- value: [-1.0, -1.0]\n+ |--- first > 0.0\n+ | |--- value: [1.0, 1.0]\n+ \"\"\").lstrip()\n+ assert export_text(reg, decimals=1,\n+ feature_names=['first']) == expected_report\n+ assert export_text(reg, decimals=1, show_weights=True,\n+ feature_names=['first']) == expected_report\n+\n \n def test_plot_tree_entropy(pyplot):\n # mostly smoke tests\n", - "fail_to_pass": [ - "sklearn/tree/tests/test_export.py::test_export_text" - ], + "fail_to_pass": ["sklearn/tree/tests/test_export.py::test_export_text"], "pass_to_pass": [ "sklearn/tree/tests/test_export.py::test_graphviz_toy", "sklearn/tree/tests/test_export.py::test_graphviz_errors", @@ -2536,9 +2494,7 @@ "sklearn/tree/tests/test_export.py::test_export_text_errors" ], "test_cmd": "pytest -rA", - "test_directives": [ - "sklearn/tree/tests/test_export.py" - ], + "test_directives": ["sklearn/tree/tests/test_export.py"], "test_runtime_ms": 1224, "weight_bucket": "medium" }, @@ -2745,9 +2701,7 @@ "sklearn/compose/tests/test_column_transformer.py::test_raise_error_if_index_not_aligned" ], "test_cmd": "pytest -rA", - "test_directives": [ - "sklearn/compose/tests/test_column_transformer.py" - ], + "test_directives": ["sklearn/compose/tests/test_column_transformer.py"], "test_runtime_ms": 214, "weight_bucket": "light" }, @@ -2796,9 +2750,7 @@ "tests/test_ext_autodoc_configs.py::test_autodoc_default_options_with_values" ], "test_cmd": "tox --current-env -epy39 -v --", - "test_directives": [ - "tests/test_ext_autodoc_configs.py" - ], + "test_directives": ["tests/test_ext_autodoc_configs.py"], "test_runtime_ms": 3362, "weight_bucket": "heavy" }, @@ -2849,9 +2801,7 @@ "tests/test_markup.py::test_default_role2" ], "test_cmd": "tox --current-env -epy39 -v --", - "test_directives": [ - "tests/test_markup.py" - ], + "test_directives": ["tests/test_markup.py"], "test_runtime_ms": 770, "weight_bucket": "light" }, @@ -2864,9 +2814,7 @@ "env_key": "sweb.env.py.x86_64.c795f4b88616b8462021ed:latest", "problem_statement": "evalf does not call _imp_ recursively\nExample from https://stackoverflow.com/questions/41818842/why-cant-i-evaluate-a-composition-of-implemented-functions-in-sympy-at-a-point:\r\n\r\n```\r\n>>> from sympy.utilities.lambdify import implemented_function\r\n>>> f = implemented_function('f', lambda x: x ** 2)\r\n>>> g = implemented_function('g', lambda x: 2 * x)\r\n>>> print(f( 2 ).evalf())\r\n4.00000000000000\r\n>>> print( g(2) .evalf())\r\n4.00000000000000\r\n>>> print(f(g(2)).evalf())\r\nf(g(2))\r\n```\r\n\r\nThe code for this is in `Function._eval_evalf`. It isn't calling evalf recursively on the return of `_imp_`. \n", "test_patch": "diff --git a/sympy/utilities/tests/test_lambdify.py b/sympy/utilities/tests/test_lambdify.py\n--- a/sympy/utilities/tests/test_lambdify.py\n+++ b/sympy/utilities/tests/test_lambdify.py\n@@ -751,6 +751,9 @@ def test_issue_2790():\n assert lambdify((x, (y, (w, z))), w + x + y + z)(1, (2, (3, 4))) == 10\n assert lambdify(x, x + 1, dummify=False)(1) == 2\n \n+def test_issue_12092():\n+ f = implemented_function('f', lambda x: x**2)\n+ assert f(f(2)).evalf() == Float(16)\n \n def test_ITE():\n assert lambdify((x, y, z), ITE(x, y, z))(True, 5, 3) == 5\n", - "fail_to_pass": [ - "test_issue_12092" - ], + "fail_to_pass": ["test_issue_12092"], "pass_to_pass": [ "test_no_args", "test_single_arg", @@ -2913,9 +2861,7 @@ "test_Min_Max" ], "test_cmd": "PYTHONWARNINGS='ignore::UserWarning,ignore::SyntaxWarning' bin/test -C --verbose", - "test_directives": [ - "sympy/utilities/tests/test_lambdify.py" - ], + "test_directives": ["sympy/utilities/tests/test_lambdify.py"], "test_runtime_ms": 2113, "weight_bucket": "heavy" }, @@ -2928,9 +2874,7 @@ "env_key": "sweb.env.py.x86_64.c795f4b88616b8462021ed:latest", "problem_statement": "Behavior of Matrix hstack and vstack changed in sympy 1.1\nIn sympy 1.0:\r\n```\r\nimport sympy as sy\r\nM1 = sy.Matrix.zeros(0, 0)\r\nM2 = sy.Matrix.zeros(0, 1)\r\nM3 = sy.Matrix.zeros(0, 2)\r\nM4 = sy.Matrix.zeros(0, 3)\r\nsy.Matrix.hstack(M1, M2, M3, M4).shape\r\n```\r\nreturns \r\n`(0, 6)`\r\n\r\nNow, same in sympy 1.1:\r\n```\r\nimport sympy as sy\r\nM1 = sy.Matrix.zeros(0, 0)\r\nM2 = sy.Matrix.zeros(0, 1)\r\nM3 = sy.Matrix.zeros(0, 2)\r\nM4 = sy.Matrix.zeros(0, 3)\r\nsy.Matrix.hstack(M1, M2, M3, M4).shape\r\n```\r\nreturns\r\n`(0, 3)\r\n`\r\nwhereas:\r\n```\r\nimport sympy as sy\r\nM1 = sy.Matrix.zeros(1, 0)\r\nM2 = sy.Matrix.zeros(1, 1)\r\nM3 = sy.Matrix.zeros(1, 2)\r\nM4 = sy.Matrix.zeros(1, 3)\r\nsy.Matrix.hstack(M1, M2, M3, M4).shape\r\n```\r\nreturns\r\n`(1, 6)\r\n`\n", "test_patch": "diff --git a/sympy/matrices/tests/test_sparse.py b/sympy/matrices/tests/test_sparse.py\n--- a/sympy/matrices/tests/test_sparse.py\n+++ b/sympy/matrices/tests/test_sparse.py\n@@ -26,6 +26,12 @@ def sparse_zeros(n):\n assert type(a.row_join(b)) == type(a)\n assert type(a.col_join(b)) == type(a)\n \n+ # make sure 0 x n matrices get stacked correctly\n+ sparse_matrices = [SparseMatrix.zeros(0, n) for n in range(4)]\n+ assert SparseMatrix.hstack(*sparse_matrices) == Matrix(0, 6, [])\n+ sparse_matrices = [SparseMatrix.zeros(n, 0) for n in range(4)]\n+ assert SparseMatrix.vstack(*sparse_matrices) == Matrix(6, 0, [])\n+\n # test element assignment\n a = SparseMatrix((\n (1, 0),\n", - "fail_to_pass": [ - "test_sparse_matrix" - ], + "fail_to_pass": ["test_sparse_matrix"], "pass_to_pass": [ "test_transpose", "test_trace", @@ -2943,9 +2887,7 @@ "test_sparse_solve" ], "test_cmd": "PYTHONWARNINGS='ignore::UserWarning,ignore::SyntaxWarning' bin/test -C --verbose", - "test_directives": [ - "sympy/matrices/tests/test_sparse.py" - ], + "test_directives": ["sympy/matrices/tests/test_sparse.py"], "test_runtime_ms": 1907, "weight_bucket": "medium" }, @@ -2958,9 +2900,7 @@ "env_key": "sweb.env.py.x86_64.c795f4b88616b8462021ed:latest", "problem_statement": "`len` of rank-0 arrays returns 0\n`sympy.tensor.array.NDimArray.__len__` always returns zero for rank-0 arrays (scalars). I believe the correct value should be one, which is the number of elements of the iterator and the observed behaviour in numpy.\r\n\r\n```python\r\n>>> import sympy\r\n>>> a = sympy.Array(3)\r\n>>> len(a)\r\n0\r\n>>> len(list(a))\r\n1\r\n```\r\nIn numpy we have the following: \r\n\r\n```python\r\n>>> import numpy\r\n>>> numpy.asarray(1).size\r\n1\r\n```\r\n\r\nThis was tested in sympy 1.2-rc1 running in Python 3.6.6\n`len` of rank-0 arrays returns 0\n`sympy.tensor.array.NDimArray.__len__` always returns zero for rank-0 arrays (scalars). I believe the correct value should be one, which is the number of elements of the iterator and the observed behaviour in numpy.\r\n\r\n```python\r\n>>> import sympy\r\n>>> a = sympy.Array(3)\r\n>>> len(a)\r\n0\r\n>>> len(list(a))\r\n1\r\n```\r\nIn numpy we have the following: \r\n\r\n```python\r\n>>> import numpy\r\n>>> numpy.asarray(1).size\r\n1\r\n```\r\n\r\nThis was tested in sympy 1.2-rc1 running in Python 3.6.6\n", "test_patch": "diff --git a/sympy/tensor/array/tests/test_immutable_ndim_array.py b/sympy/tensor/array/tests/test_immutable_ndim_array.py\n--- a/sympy/tensor/array/tests/test_immutable_ndim_array.py\n+++ b/sympy/tensor/array/tests/test_immutable_ndim_array.py\n@@ -9,6 +9,10 @@\n \n \n def test_ndim_array_initiation():\n+ arr_with_no_elements = ImmutableDenseNDimArray([], shape=(0,))\n+ assert len(arr_with_no_elements) == 0\n+ assert arr_with_no_elements.rank() == 1\n+\n arr_with_one_element = ImmutableDenseNDimArray([23])\n assert len(arr_with_one_element) == 1\n assert arr_with_one_element[0] == 23\n@@ -73,11 +77,11 @@ def test_ndim_array_initiation():\n \n from sympy.abc import x\n rank_zero_array = ImmutableDenseNDimArray(x)\n- assert len(rank_zero_array) == 0\n+ assert len(rank_zero_array) == 1\n assert rank_zero_array.shape == ()\n assert rank_zero_array.rank() == 0\n assert rank_zero_array[()] == x\n- raises(ValueError, lambda: rank_zero_array[0])\n+ assert rank_zero_array[0] == x\n \n \n def test_reshape():\n", - "fail_to_pass": [ - "test_ndim_array_initiation" - ], + "fail_to_pass": ["test_ndim_array_initiation"], "pass_to_pass": [ "test_reshape", "test_iterator", @@ -2978,9 +2918,7 @@ "test_symbolic_indexing" ], "test_cmd": "PYTHONWARNINGS='ignore::UserWarning,ignore::SyntaxWarning' bin/test -C --verbose", - "test_directives": [ - "sympy/tensor/array/tests/test_immutable_ndim_array.py" - ], + "test_directives": ["sympy/tensor/array/tests/test_immutable_ndim_array.py"], "test_runtime_ms": 1447, "weight_bucket": "medium" }, @@ -2993,9 +2931,7 @@ "env_key": "sweb.env.py.x86_64.c795f4b88616b8462021ed:latest", "problem_statement": "Factor with extension=True drops a factor of y-1\nI guess this related (or a duplicate of?) #5786\r\n\r\nThis is from stackoverflow:\r\nhttps://stackoverflow.com/questions/60682765/python-sympy-factoring-polynomial-over-complex-numbers\r\n```julia\r\nIn [9]: z = expand((x-1)*(y-1)) \r\n\r\nIn [10]: z \r\nOut[10]: x\u22c5y - x - y + 1\r\n\r\nIn [11]: factor(z) \r\nOut[11]: (x - 1)\u22c5(y - 1)\r\n\r\nIn [12]: factor(z, extension=[I]) \r\nOut[12]: x - 1\r\n```\nFactor with extension=True drops a factor of y-1\n\r\nFactor with extension=True drops a factor of y-1\r\n#### References to other Issues or PRs\r\n\r\nFixes #18895 \r\n\r\n#### Brief description of what is fixed or changed\r\n\r\n\r\n#### Other comments\r\n\r\n\r\n#### Release Notes\r\n\r\n\r\n\r\n\r\nNO ENTRY\r\n\n", "test_patch": "diff --git a/sympy/polys/tests/test_polytools.py b/sympy/polys/tests/test_polytools.py\n--- a/sympy/polys/tests/test_polytools.py\n+++ b/sympy/polys/tests/test_polytools.py\n@@ -58,7 +58,7 @@\n from sympy.core.basic import _aresame\n from sympy.core.compatibility import iterable\n from sympy.core.mul import _keep_coeff\n-from sympy.testing.pytest import raises, XFAIL, warns_deprecated_sympy\n+from sympy.testing.pytest import raises, warns_deprecated_sympy\n \n from sympy.abc import a, b, c, d, p, q, t, w, x, y, z\n from sympy import MatrixSymbol, Matrix\n@@ -3249,7 +3249,6 @@ def test_poly_matching_consistency():\n assert Poly(x, x) * I == Poly(I*x, x)\n \n \n-@XFAIL\n def test_issue_5786():\n assert expand(factor(expand(\n (x - I*y)*(z - I*t)), extension=[I])) == -I*t*x - t*y + x*z - I*y*z\n", - "fail_to_pass": [ - "test_issue_5786" - ], + "fail_to_pass": ["test_issue_5786"], "pass_to_pass": [ "test_Poly_mixed_operations", "test_Poly_from_dict", @@ -3147,9 +3083,7 @@ "test_issue_18205" ], "test_cmd": "PYTHONWARNINGS='ignore::UserWarning,ignore::SyntaxWarning' bin/test -C --verbose", - "test_directives": [ - "sympy/polys/tests/test_polytools.py" - ], + "test_directives": ["sympy/polys/tests/test_polytools.py"], "test_runtime_ms": 14786, "weight_bucket": "heavy" }, @@ -3162,9 +3096,7 @@ "env_key": "sweb.env.py.x86_64.c795f4b88616b8462021ed:latest", "problem_statement": "Result from clear_denoms() prints like zero poly but behaves wierdly (due to unstripped DMP)\nThe was the immediate cause of the ZeroDivisionError in #17990.\r\n\r\nCalling `clear_denoms()` on a complicated constant poly that turns out to be zero:\r\n\r\n```\r\n>>> from sympy import *\r\n>>> x = symbols(\"x\")\r\n>>> f = Poly(sympify(\"-117968192370600*18**(1/3)/(217603955769048*(24201 + 253*sqrt(9165))**(1/3) + 2273005839412*sqrt(9165)*(24201 + 253*sqrt(9165))**(1/3)) - 15720318185*2**(2/3)*3**(1/3)*(24201 + 253*sqrt(9165))**(2/3)/(217603955769048*(24201 + 253*sqrt(9165))**(1/3) + 2273005839412*sqrt(9165)*(24201 + 253*sqrt(9165))**(1/3)) + 15720318185*12**(1/3)*(24201 + 253*sqrt(9165))**(2/3)/(217603955769048*(24201 + 253*sqrt(9165))**(1/3) + 2273005839412*sqrt(9165)*(24201 + 253*sqrt(9165))**(1/3)) + 117968192370600*2**(1/3)*3**(2/3)/(217603955769048*(24201 + 253*sqrt(9165))**(1/3) + 2273005839412*sqrt(9165)*(24201 + 253*sqrt(9165))**(1/3))\"), x)\r\n>>> coeff, bad_poly = f.clear_denoms()\r\n>>> coeff\r\n(217603955769048*(24201 + 253*sqrt(9165))**(1/3) + 2273005839412*sqrt(9165)*(24201 + 253*sqrt(9165))**(1/3)\r\n>>> bad_poly\r\nPoly(0, x, domain='EX'))\r\n```\r\n\r\nThe result prints like the zero polynomial but behaves inconsistently:\r\n\r\n```\r\n>>> bad_poly\r\nPoly(0, x, domain='EX')\r\n>>> bad_poly.is_zero\r\nFalse\r\n>>> bad_poly.as_expr()\r\n0\r\n>>> _.is_zero\r\nTrue\r\n```\r\n\r\n~~There may be valid cases (at least with EX coefficients) where the two valued Poly.is_zero is False but as_expr() evaluates to 0~~ (@jksuom points out this is a bug in #20428), but other Poly methods don't handle `bad_poly` very well.\r\n\r\ne.g.\r\n\r\n```\r\n>>> Poly(0, x).terms_gcd()\r\n((0,), Poly(0, x, domain='ZZ'))\r\n>>> bad_poly.terms_gcd()\r\nTraceback (most recent call last):\r\n File \"\", line 1, in \r\n File \"/Users/ehren/Documents/esym26/sympy/polys/polytools.py\", line 1227, in terms_gcd\r\n J, result = f.rep.terms_gcd()\r\n File \"/Users/ehren/Documents/esym26/sympy/polys/polyclasses.py\", line 410, in terms_gcd\r\n J, F = dmp_terms_gcd(f.rep, f.lev, f.dom)\r\n File \"/Users/ehren/Documents/esym26/sympy/polys/densebasic.py\", line 1681, in dmp_terms_gcd\r\n G = monomial_min(*list(F.keys()))\r\n File \"/Users/ehren/Documents/esym26/sympy/polys/monomials.py\", line 359, in monomial_min\r\n M = list(monoms[0])\r\nIndexError: tuple index out of range\r\n```\r\n\r\nAlso sometime in the last year Poly.primitive has been changed to slightly better handle this bad poly.\r\n\r\n```\r\n>>> Poly(0, x).primitive()\r\n(0, Poly(0, x, domain='ZZ'))\r\n>>> bad_poly.primitive()\r\n(1, Poly(0, x, domain='EX'))\r\n```\r\n\r\nbut in earlier versions of SymPy:\r\n\r\n```\r\n>>> bad_poly.primitive()\r\nTraceback (most recent call last):\r\n File \"\", line 1, in \r\n File \"/Users/ehren/Documents/esym7/sympy/polys/polytools.py\", line 2986, in primitive\r\n cont, result = f.rep.primitive()\r\n File \"/Users/ehren/Documents/esym7/sympy/polys/polyclasses.py\", line 722, in primitive\r\n cont, F = dmp_ground_primitive(f.rep, f.lev, f.dom)\r\n File \"/Users/ehren/Documents/esym7/sympy/polys/densetools.py\", line 715, in dmp_ground_primitive\r\n return dup_primitive(f, K)\r\n File \"/Users/ehren/Documents/esym7/sympy/polys/densetools.py\", line 689, in dup_primitive\r\n return cont, dup_quo_ground(f, cont, K)\r\n File \"/Users/ehren/Documents/esym7/sympy/polys/densearith.py\", line 317, in dup_quo_ground\r\n raise ZeroDivisionError('polynomial division')\r\n```\r\n\r\nwhich was the cause of the ZeroDivisionError reported in #17990.\r\n\r\nLooking at the underlying DMP, there is an unstripped leading 0 in the list representation of the Poly\r\n\r\n```\r\n>>> bad_poly.rep\r\nDMP([EX(0)], EX, None)\r\n```\r\n\r\nwhich should be\r\n\r\n```\r\n>>> Poly(0, x, domain=\"EX\").rep\r\nDMP([], EX, None)\r\n```\n", "test_patch": "diff --git a/sympy/polys/tests/test_polytools.py b/sympy/polys/tests/test_polytools.py\n--- a/sympy/polys/tests/test_polytools.py\n+++ b/sympy/polys/tests/test_polytools.py\n@@ -1458,6 +1458,20 @@ def test_Poly_rat_clear_denoms():\n assert f.rat_clear_denoms(g) == (f, g)\n \n \n+def test_issue_20427():\n+ f = Poly(-117968192370600*18**(S(1)/3)/(217603955769048*(24201 +\n+ 253*sqrt(9165))**(S(1)/3) + 2273005839412*sqrt(9165)*(24201 +\n+ 253*sqrt(9165))**(S(1)/3)) - 15720318185*2**(S(2)/3)*3**(S(1)/3)*(24201\n+ + 253*sqrt(9165))**(S(2)/3)/(217603955769048*(24201 + 253*sqrt(9165))**\n+ (S(1)/3) + 2273005839412*sqrt(9165)*(24201 + 253*sqrt(9165))**(S(1)/3))\n+ + 15720318185*12**(S(1)/3)*(24201 + 253*sqrt(9165))**(S(2)/3)/(\n+ 217603955769048*(24201 + 253*sqrt(9165))**(S(1)/3) + 2273005839412*\n+ sqrt(9165)*(24201 + 253*sqrt(9165))**(S(1)/3)) + 117968192370600*2**(\n+ S(1)/3)*3**(S(2)/3)/(217603955769048*(24201 + 253*sqrt(9165))**(S(1)/3)\n+ + 2273005839412*sqrt(9165)*(24201 + 253*sqrt(9165))**(S(1)/3)), x)\n+ assert f == Poly(0, x, domain='EX')\n+\n+\n def test_Poly_integrate():\n assert Poly(x + 1).integrate() == Poly(x**2/2 + x)\n assert Poly(x + 1).integrate(x) == Poly(x**2/2 + x)\n", - "fail_to_pass": [ - "test_issue_20427" - ], + "fail_to_pass": ["test_issue_20427"], "pass_to_pass": [ "test_Poly_mixed_operations", "test_Poly_from_dict", @@ -3322,9 +3254,7 @@ "test_poly_copy_equals_original" ], "test_cmd": "PYTHONWARNINGS='ignore::UserWarning,ignore::SyntaxWarning' bin/test -C --verbose", - "test_directives": [ - "sympy/polys/tests/test_polytools.py" - ], + "test_directives": ["sympy/polys/tests/test_polytools.py"], "test_runtime_ms": 14498, "weight_bucket": "heavy" }, @@ -3337,9 +3267,7 @@ "env_key": "sweb.env.py.x86_64.c795f4b88616b8462021ed:latest", "problem_statement": "Python code printer not respecting tuple with one element\nHi,\r\n\r\nThanks for the recent updates in SymPy! I'm trying to update my code to use SymPy 1.10 but ran into an issue with the Python code printer. MWE:\r\n\r\n\r\n```python\r\nimport inspect\r\nfrom sympy import lambdify\r\n\r\ninspect.getsource(lambdify([], tuple([1])))\r\n```\r\nSymPy 1.9 and under outputs:\r\n```\r\n'def _lambdifygenerated():\\n return (1,)\\n'\r\n```\r\n\r\nBut SymPy 1.10 gives\r\n\r\n```\r\n'def _lambdifygenerated():\\n return (1)\\n'\r\n```\r\nNote the missing comma after `1` that causes an integer to be returned instead of a tuple. \r\n\r\nFor tuples with two or more elements, the generated code is correct:\r\n```python\r\ninspect.getsource(lambdify([], tuple([1, 2])))\r\n```\r\nIn SymPy 1.10 and under, outputs:\r\n\r\n```\r\n'def _lambdifygenerated():\\n return (1, 2)\\n'\r\n```\r\nThis result is expected.\r\n\r\nNot sure if this is a regression. As this breaks my program which assumes the return type to always be a tuple, could you suggest a workaround from the code generation side? Thank you. \n", "test_patch": "diff --git a/sympy/utilities/tests/test_lambdify.py b/sympy/utilities/tests/test_lambdify.py\n--- a/sympy/utilities/tests/test_lambdify.py\n+++ b/sympy/utilities/tests/test_lambdify.py\n@@ -1192,6 +1192,8 @@ def test_issue_14941():\n # test tuple\n f2 = lambdify([x, y], (y, x), 'sympy')\n assert f2(2, 3) == (3, 2)\n+ f2b = lambdify([], (1,)) # gh-23224\n+ assert f2b() == (1,)\n \n # test list\n f3 = lambdify([x, y], [y, x], 'sympy')\n", - "fail_to_pass": [ - "test_issue_14941" - ], + "fail_to_pass": ["test_issue_14941"], "pass_to_pass": [ "test_no_args", "test_single_arg", @@ -3404,9 +3332,7 @@ "test_lambdify_cse" ], "test_cmd": "PYTHONWARNINGS='ignore::UserWarning,ignore::SyntaxWarning' bin/test -C --verbose", - "test_directives": [ - "sympy/utilities/tests/test_lambdify.py" - ], + "test_directives": ["sympy/utilities/tests/test_lambdify.py"], "test_runtime_ms": 4000, "weight_bucket": "heavy" } diff --git a/experiments/test/counting-backend.test.ts b/experiments/test/counting-backend.test.ts index 892c8a3..85a6f7b 100644 --- a/experiments/test/counting-backend.test.ts +++ b/experiments/test/counting-backend.test.ts @@ -1,24 +1,23 @@ -import { describe, it, expect } from "vitest"; -import type { SessionStorageBackend } from "@earendil-works/pi-coding-agent"; -import { CountingBackend } from "../src/counting-backend"; +import { describe, it, expect } from 'vitest'; +import type { SessionStorageBackend } from '@earendil-works/pi-coding-agent'; +import { CountingBackend } from '../src/counting-backend'; // A trivial in-memory backend returning a fixed list, to exercise the decorator. function fakeBackend(entries: unknown[]): SessionStorageBackend { return { append: async () => {}, - read: async (_sid: string, fromPosition = 1) => - entries.slice(fromPosition - 1) as never, + read: async (_sid: string, fromPosition = 1) => entries.slice(fromPosition - 1) as never, latestCheckpoint: async () => null, list: async () => [], }; } -describe("CountingBackend", () => { - it("tallies entries and bytes per read, and reset() zeroes", async () => { +describe('CountingBackend', () => { + it('tallies entries and bytes per read, and reset() zeroes', async () => { const entries = [{ a: 1 }, { b: 22 }, { c: 333 }]; const cb = new CountingBackend(fakeBackend(entries)); - const full = await cb.read("s"); + const full = await cb.read('s'); expect(full.length).toBe(3); const c1 = cb.counts(); expect(c1.reads).toBe(1); @@ -27,7 +26,7 @@ describe("CountingBackend", () => { entries.reduce((n, e) => n + Buffer.byteLength(JSON.stringify(e)), 0), ); - await cb.read("s", 3); // tail of 1 entry + await cb.read('s', 3); // tail of 1 entry expect(cb.counts().reads).toBe(2); expect(cb.counts().entriesRead).toBe(4); @@ -35,9 +34,9 @@ describe("CountingBackend", () => { expect(cb.counts()).toEqual({ reads: 0, entriesRead: 0, bytesRead: 0, checkpointLookups: 0 }); }); - it("counts latestCheckpoint lookups separately and delegates", async () => { + it('counts latestCheckpoint lookups separately and delegates', async () => { const cb = new CountingBackend(fakeBackend([])); - expect(await cb.latestCheckpoint("s")).toBeNull(); + expect(await cb.latestCheckpoint('s')).toBeNull(); expect(cb.counts().checkpointLookups).toBe(1); expect(cb.counts().entriesRead).toBe(0); }); diff --git a/experiments/test/e2-reconstruction-cost.test.ts b/experiments/test/e2-reconstruction-cost.test.ts index bf4ce28..0ca51ae 100644 --- a/experiments/test/e2-reconstruction-cost.test.ts +++ b/experiments/test/e2-reconstruction-cost.test.ts @@ -1,20 +1,15 @@ -import { describe, it, expect, afterAll } from "vitest"; -import { mkdirSync, readFileSync, writeFileSync } from "node:fs"; -import { join, resolve } from "node:path"; -import { fileURLToPath } from "node:url"; -import { SessionManager, type FileEntry } from "@earendil-works/pi-coding-agent"; -import { RedisSessionBackend } from "@sh/session-backend"; -import { BufferedRedisBackend } from "@sh/harness/buffered-redis-backend"; -import { CountingBackend } from "../src/counting-backend"; -import { buildCompactedSession } from "../src/session-fixture"; -import { - buildResultsMarkdown, - deterministicView, - parseE2Table, - type E2Row, -} from "../src/report"; - -const REDIS = process.env.REDIS_URL ?? "redis://127.0.0.1:6379"; +import { describe, it, expect, afterAll } from 'vitest'; +import { mkdirSync, readFileSync, writeFileSync } from 'node:fs'; +import { join, resolve } from 'node:path'; +import { fileURLToPath } from 'node:url'; +import { SessionManager, type FileEntry } from '@earendil-works/pi-coding-agent'; +import { RedisSessionBackend } from '@sh/session-backend'; +import { BufferedRedisBackend } from '@sh/harness/buffered-redis-backend'; +import { CountingBackend } from '../src/counting-backend'; +import { buildCompactedSession } from '../src/session-fixture'; +import { buildResultsMarkdown, deterministicView, parseE2Table, type E2Row } from '../src/report'; + +const REDIS = process.env.REDIS_URL ?? 'redis://127.0.0.1:6379'; const store = new RedisSessionBackend(REDIS); const sids: string[] = []; const Ns = [50, 200, 1000, 5000]; @@ -51,8 +46,8 @@ async function measure(sessionId: string): Promise<{ }; } -describe("E2 — reconstruction cost", () => { - it("checkpoint read stays ~constant while backend grows; ratio increases with N", async () => { +describe('E2 — reconstruction cost', () => { + it('checkpoint read stays ~constant while backend grows; ratio increases with N', async () => { const rows: E2Row[] = []; for (const n of Ns) { const fx = await buildCompactedSession(store, { n, tailKept: 4 }); @@ -95,20 +90,20 @@ describe("E2 — reconstruction cost", () => { // machine-local wall-clock timings. Override the directory with SH_E2_RESULTS_DIR. const outDir = process.env.SH_E2_RESULTS_DIR ? resolve(process.env.SH_E2_RESULTS_DIR) - : fileURLToPath(new URL("../.results", import.meta.url)); + : fileURLToPath(new URL('../.results', import.meta.url)); mkdirSync(outDir, { recursive: true }); - writeFileSync(join(outDir, "RESULTS.md"), report); + writeFileSync(join(outDir, 'RESULTS.md'), report); // The committed RESULTS.md is a checked-in baseline: assert the reproducible columns // still match it, so a change that moves the read counts has to be acknowledged rather // than quietly rewriting the recorded result. Only entries + ratio are compared -- // backendBytes is environment-sensitive (+4 in CI) and the ms columns vary per run. // Refresh deliberately with SH_E2_UPDATE_BASELINE=1 when a change legitimately moves them. - const baselinePath = fileURLToPath(new URL("../RESULTS.md", import.meta.url)); - if (process.env.SH_E2_UPDATE_BASELINE === "1") { + const baselinePath = fileURLToPath(new URL('../RESULTS.md', import.meta.url)); + if (process.env.SH_E2_UPDATE_BASELINE === '1') { writeFileSync(baselinePath, report); } else { - const baseline = parseE2Table(readFileSync(baselinePath, "utf8")); + const baseline = parseE2Table(readFileSync(baselinePath, 'utf8')); expect(deterministicView(rows)).toEqual(deterministicView(baseline)); } diff --git a/experiments/test/e5-budget-live.test.ts b/experiments/test/e5-budget-live.test.ts index 05485ce..e1ad94e 100644 --- a/experiments/test/e5-budget-live.test.ts +++ b/experiments/test/e5-budget-live.test.ts @@ -1,10 +1,10 @@ -import { describe, it, expect, afterAll } from "vitest"; -import { type FileEntry } from "@earendil-works/pi-coding-agent"; -import { RedisSessionBackend } from "@sh/session-backend"; -import { runTurn } from "@sh/harness/run-turn"; +import { describe, it, expect, afterAll } from 'vitest'; +import { type FileEntry } from '@earendil-works/pi-coding-agent'; +import { RedisSessionBackend } from '@sh/session-backend'; +import { runTurn } from '@sh/harness/run-turn'; -const REDIS = process.env.REDIS_URL ?? "redis://127.0.0.1:6379"; -const LIVE = process.env.SH_RUN_LIVE === "1" && !!process.env.ANTHROPIC_AUTH_TOKEN; +const REDIS = process.env.REDIS_URL ?? 'redis://127.0.0.1:6379'; +const LIVE = process.env.SH_RUN_LIVE === '1' && !!process.env.ANTHROPIC_AUTH_TOKEN; const store = new RedisSessionBackend(REDIS); const sids: string[] = []; @@ -16,21 +16,22 @@ afterAll(async () => { async function abortCount(sessionId: string): Promise { const entries = await store.read(sessionId); return entries.filter( - (r) => (r.entry as { type?: string }).type === "custom" && - (r.entry as { customType?: string }).customType === "abort", + (r) => + (r.entry as { type?: string }).type === 'custom' && + (r.entry as { customType?: string }).customType === 'abort', ).length; } -describe("E5 — budget voter enforcement (live)", () => { +describe('E5 — budget voter enforcement (live)', () => { // Skips unless SH_RUN_LIVE=1 and a key is present (see README). (LIVE ? it : it.skip)( - "blocks a real tool call and records exactly one abort under a tiny cap", + 'blocks a real tool call and records exactly one abort under a tiny cap', async () => { // SH_BUDGET_TOKENS=1 must be exported by the runner so run-turn registers the voter. expect(Number(process.env.SH_BUDGET_TOKENS)).toBeGreaterThan(0); // A prompt that forces a tool call (a tool must be registered — see README §tools). const prompt = - "Use the shell tool to run `echo hello`. You must call a tool; do not answer directly."; + 'Use the shell tool to run `echo hello`. You must call a tool; do not answer directly.'; const result = await runTurn(prompt, undefined, { redisUrl: REDIS }); sids.push(result.sessionId); diff --git a/experiments/test/e5-budget-structural.test.ts b/experiments/test/e5-budget-structural.test.ts index f4a684d..dd1815a 100644 --- a/experiments/test/e5-budget-structural.test.ts +++ b/experiments/test/e5-budget-structural.test.ts @@ -1,10 +1,10 @@ -import { describe, it, expect, afterAll } from "vitest"; -import { SessionManager, type FileEntry } from "@earendil-works/pi-coding-agent"; -import { RedisSessionBackend } from "@sh/session-backend"; -import { BufferedRedisBackend } from "@sh/harness/buffered-redis-backend"; -import { budgetVoterExtension } from "@sh/harness/budget-voter"; +import { describe, it, expect, afterAll } from 'vitest'; +import { SessionManager, type FileEntry } from '@earendil-works/pi-coding-agent'; +import { RedisSessionBackend } from '@sh/session-backend'; +import { BufferedRedisBackend } from '@sh/harness/buffered-redis-backend'; +import { budgetVoterExtension } from '@sh/harness/budget-voter'; -const REDIS = process.env.REDIS_URL ?? "redis://127.0.0.1:6379"; +const REDIS = process.env.REDIS_URL ?? 'redis://127.0.0.1:6379'; const store = new RedisSessionBackend(REDIS); const sids: string[] = []; @@ -20,7 +20,15 @@ function spendCtx(total: number) { getBranch: () => total === 0 ? [] - : [{ type: "message", message: { role: "assistant", usage: { input: total, output: 0, cacheRead: 0, cacheWrite: 0 } } }], + : [ + { + type: 'message', + message: { + role: 'assistant', + usage: { input: total, output: 0, cacheRead: 0, cacheWrite: 0 }, + }, + }, + ], }, } as never; } @@ -29,7 +37,11 @@ function spendCtx(total: number) { // (the headless path does not emit it). function register(sm: SessionManager, limit: number) { const handlers: Record unknown> = {}; - const pi = { on: (ev: string, h: (e: unknown, ctx: unknown) => unknown) => { handlers[ev] = h; } }; + const pi = { + on: (ev: string, h: (e: unknown, ctx: unknown) => unknown) => { + handlers[ev] = h; + }, + }; budgetVoterExtension(sm, { limit, baseline: 0 })(pi as never); return handlers; } @@ -37,13 +49,14 @@ function register(sm: SessionManager, limit: number) { async function abortCount(sessionId: string): Promise { const entries = await store.read(sessionId); return entries.filter( - (r) => (r.entry as { type?: string; customType?: string }).type === "custom" && - (r.entry as { customType?: string }).customType === "abort", + (r) => + (r.entry as { type?: string; customType?: string }).type === 'custom' && + (r.entry as { customType?: string }).customType === 'abort', ).length; } -describe("E5 — budget voter enforcement (structural, real Redis)", () => { - it("blocks the tool call and persists exactly one abort entry once over cap", async () => { +describe('E5 — budget voter enforcement (structural, real Redis)', () => { + it('blocks the tool call and persists exactly one abort entry once over cap', async () => { const backend = new BufferedRedisBackend(store); const sm = SessionManager.create(process.cwd(), undefined, undefined, backend); const sid = sm.getSessionId(); @@ -52,12 +65,12 @@ describe("E5 — budget voter enforcement (structural, real Redis)", () => { const handlers = register(sm, 50); const res = handlers.tool_call({}, spendCtx(60)); // 60 > 50, baseline 0, no session_start - expect(res).toEqual({ block: true, reason: "Session token budget exceeded" }); + expect(res).toEqual({ block: true, reason: 'Session token budget exceeded' }); await backend.flush(); expect(await abortCount(sid)).toBe(1); }); - it("is inert when the cap is disabled (limit <= 0): no block, no abort", async () => { + it('is inert when the cap is disabled (limit <= 0): no block, no abort', async () => { const backend = new BufferedRedisBackend(store); const sm = SessionManager.create(process.cwd(), undefined, undefined, backend); const sid = sm.getSessionId(); diff --git a/experiments/test/e6-saturation-structural.test.ts b/experiments/test/e6-saturation-structural.test.ts index 792c085..6ec0333 100644 --- a/experiments/test/e6-saturation-structural.test.ts +++ b/experiments/test/e6-saturation-structural.test.ts @@ -1,4 +1,4 @@ -import { describe, it, expect } from "vitest"; +import { describe, it, expect } from 'vitest'; import { detectKnee, dutyCycle, @@ -7,25 +7,25 @@ import { buildRatioCurve, type LadderPoint, type WorkloadPoint, -} from "../src/sharing"; +} from '../src/sharing'; -describe("E6 — saturation analysis (structural)", () => { - it("computes duty cycle and derived ratio", () => { +describe('E6 — saturation analysis (structural)', () => { + it('computes duty cycle and derived ratio', () => { expect(dutyCycle(250, 1000)).toBeCloseTo(0.25, 5); expect(derivedRatio(0.25)).toBe(4.0); expect(() => dutyCycle(1, 0)).toThrow(); expect(() => derivedRatio(0)).toThrow(); }); - it("caps duty cycle at 1 and enforces the sanity floor", () => { + it('caps duty cycle at 1 and enforces the sanity floor', () => { expect(dutyCycle(1500, 1000)).toBe(1); expect(sanityFloorPass(8, 4)).toBe(true); expect(sanityFloorPass(2, 4)).toBe(false); }); }); -describe("detectKnee — sustained-decline (noise-tolerant)", () => { - it("still returns the last healthy rung when the top rung blows past the latency bound", () => { +describe('detectKnee — sustained-decline (noise-tolerant)', () => { + it('still returns the last healthy rung when the top rung blows past the latency bound', () => { const series: LadderPoint[] = [ { c: 1, throughput: 1.0, p95Ms: 1000 }, { c: 2, throughput: 1.9, p95Ms: 1050 }, @@ -36,7 +36,7 @@ describe("detectKnee — sustained-decline (noise-tolerant)", () => { expect(detectKnee(series, 2)).toBe(8); }); - it("tolerates a single throughput dip within the latency bound (does not break early)", () => { + it('tolerates a single throughput dip within the latency bound (does not break early)', () => { const series: LadderPoint[] = [ { c: 1, throughput: 1.0, p95Ms: 1000 }, { c: 2, throughput: 2.0, p95Ms: 1100 }, @@ -46,7 +46,7 @@ describe("detectKnee — sustained-decline (noise-tolerant)", () => { expect(detectKnee(series, 2)).toBe(8); }); - it("breaks on a sustained decline (patience consecutive unhealthy rungs)", () => { + it('breaks on a sustained decline (patience consecutive unhealthy rungs)', () => { const series: LadderPoint[] = [ { c: 1, throughput: 1.0, p95Ms: 1000 }, { c: 2, throughput: 2.0, p95Ms: 1100 }, @@ -57,20 +57,20 @@ describe("detectKnee — sustained-decline (noise-tolerant)", () => { expect(detectKnee(series, 2)).toBe(2); }); - it("throws when there is no c=1 baseline", () => { + it('throws when there is no c=1 baseline', () => { expect(() => detectKnee([{ c: 2, throughput: 1, p95Ms: 1 }], 2)).toThrow(/baseline/); }); }); -describe("buildRatioCurve — N as a function of per-leaf sandbox work", () => { - it("maps each workload point to duty + N, N decreasing as sandbox work rises", () => { +describe('buildRatioCurve — N as a function of per-leaf sandbox work', () => { + it('maps each workload point to duty + N, N decreasing as sandbox work rises', () => { const pts: WorkloadPoint[] = [ - { label: "L0", execMs: 300, execCount: 2, wallMs: 12000 }, - { label: "L1", execMs: 1200, execCount: 6, wallMs: 12000 }, - { label: "L2", execMs: 3000, execCount: 14, wallMs: 12000 }, + { label: 'L0', execMs: 300, execCount: 2, wallMs: 12000 }, + { label: 'L1', execMs: 1200, execCount: 6, wallMs: 12000 }, + { label: 'L2', execMs: 3000, execCount: 14, wallMs: 12000 }, ]; const curve = buildRatioCurve(pts); - expect(curve.map((c) => c.label)).toEqual(["L0", "L1", "L2"]); + expect(curve.map((c) => c.label)).toEqual(['L0', 'L1', 'L2']); expect(curve[0].duty).toBeCloseTo(0.025, 3); expect(curve[0].n).toBe(40); // 1/0.025 expect(curve[0].execCount).toBe(2); @@ -79,7 +79,7 @@ describe("buildRatioCurve — N as a function of per-leaf sandbox work", () => { expect(curve[2].n).toBeLessThan(curve[0].n); }); - it("propagates the dutyCycle guard (wallMs <= 0 throws)", () => { - expect(() => buildRatioCurve([{ label: "x", execMs: 1, execCount: 1, wallMs: 0 }])).toThrow(); + it('propagates the dutyCycle guard (wallMs <= 0 throws)', () => { + expect(() => buildRatioCurve([{ label: 'x', execMs: 1, execCount: 1, wallMs: 0 }])).toThrow(); }); }); diff --git a/experiments/test/e7-converge-contention-structural.test.ts b/experiments/test/e7-converge-contention-structural.test.ts index 3f57f60..1d47585 100644 --- a/experiments/test/e7-converge-contention-structural.test.ts +++ b/experiments/test/e7-converge-contention-structural.test.ts @@ -1,40 +1,40 @@ -import { describe, it, expect, afterAll } from "vitest"; -import { worktreeConsistent, type LeafObservation } from "../src/sharing"; -import { RedisLeaseStore } from "@sh/harness/sandbox-lease"; +import { describe, it, expect, afterAll } from 'vitest'; +import { worktreeConsistent, type LeafObservation } from '../src/sharing'; +import { RedisLeaseStore } from '@sh/harness/sandbox-lease'; -describe("E7 — mixed-ref consistency (structural)", () => { - it("passes when every leaf observed its own ref, fails on cross-contamination", () => { +describe('E7 — mixed-ref consistency (structural)', () => { + it('passes when every leaf observed its own ref, fails on cross-contamination', () => { const good: LeafObservation[] = [ - { runId: "a", expectedRef: "branch-0", observedMarker: "branch-0" }, - { runId: "b", expectedRef: "branch-1", observedMarker: "branch-1" }, + { runId: 'a', expectedRef: 'branch-0', observedMarker: 'branch-0' }, + { runId: 'b', expectedRef: 'branch-1', observedMarker: 'branch-1' }, ]; expect(worktreeConsistent(good).ok).toBe(true); const bad: LeafObservation[] = [ - { runId: "a", expectedRef: "branch-0", observedMarker: "branch-0" }, - { runId: "b", expectedRef: "branch-1", observedMarker: "branch-0" }, // leaked sibling ref + { runId: 'a', expectedRef: 'branch-0', observedMarker: 'branch-0' }, + { runId: 'b', expectedRef: 'branch-1', observedMarker: 'branch-0' }, // leaked sibling ref ]; const r = worktreeConsistent(bad); expect(r.ok).toBe(false); expect(r.mismatches).toHaveLength(1); - expect(r.mismatches[0].runId).toBe("b"); + expect(r.mismatches[0].runId).toBe('b'); }); }); -const REDIS = process.env.REDIS_URL ?? "redis://127.0.0.1:6379"; -describe("E7 — lease never exceeds cap (structural, real Redis)", () => { +const REDIS = process.env.REDIS_URL ?? 'redis://127.0.0.1:6379'; +describe('E7 — lease never exceeds cap (structural, real Redis)', () => { const store = new RedisLeaseStore(REDIS); const pod = `e7-cap-test-${process.pid}`; afterAll(async () => { // best-effort cleanup of the test pod's lease set // (RedisLeaseStore has no delete-key; release each member we added) - for (const id of ["r0", "r1", "r2", "r3", "r4"]) await store.release(pod, id); + for (const id of ['r0', 'r1', 'r2', 'r3', 'r4']) await store.release(pod, id); }); - it("grants at most `cap` concurrent leases", async () => { + it('grants at most `cap` concurrent leases', async () => { const cap = 3; const results: boolean[] = []; - for (const id of ["r0", "r1", "r2", "r3", "r4"]) { + for (const id of ['r0', 'r1', 'r2', 'r3', 'r4']) { results.push(await store.acquire(pod, cap, id, 60_000)); } expect(results.filter(Boolean).length).toBe(cap); // exactly 3 granted, 2 refused diff --git a/experiments/test/predictions.test.ts b/experiments/test/predictions.test.ts index e8ca34e..8964ab1 100644 --- a/experiments/test/predictions.test.ts +++ b/experiments/test/predictions.test.ts @@ -1,9 +1,13 @@ -import { describe, it, expect } from "vitest"; -import { predictionRecord } from "../src/workload.js"; +import { describe, it, expect } from 'vitest'; +import { predictionRecord } from '../src/workload.js'; -describe("predictions.jsonl record", () => { - it("is the official SWE-bench predictions shape", () => { - const r = predictionRecord("django__django-123", "claude-haiku-4-5", "diff --git a b\n"); - expect(r).toEqual({ instance_id: "django__django-123", model_name_or_path: "claude-haiku-4-5", model_patch: "diff --git a b\n" }); +describe('predictions.jsonl record', () => { + it('is the official SWE-bench predictions shape', () => { + const r = predictionRecord('django__django-123', 'claude-haiku-4-5', 'diff --git a b\n'); + expect(r).toEqual({ + instance_id: 'django__django-123', + model_name_or_path: 'claude-haiku-4-5', + model_patch: 'diff --git a b\n', + }); }); }); diff --git a/experiments/test/report.test.ts b/experiments/test/report.test.ts index 6f44912..6e74cf4 100644 --- a/experiments/test/report.test.ts +++ b/experiments/test/report.test.ts @@ -1,7 +1,7 @@ -import { describe, it, expect } from "vitest"; -import { readFileSync } from "node:fs"; -import { fileURLToPath } from "node:url"; -import { buildResultsMarkdown, parseE2Table, deterministicView, type E2Row } from "../src/report"; +import { describe, it, expect } from 'vitest'; +import { readFileSync } from 'node:fs'; +import { fileURLToPath } from 'node:url'; +import { buildResultsMarkdown, parseE2Table, deterministicView, type E2Row } from '../src/report'; const ROWS: E2Row[] = [ { @@ -26,8 +26,8 @@ const ROWS: E2Row[] = [ }, ]; -describe("parseE2Table", () => { - it("round-trips the table that buildResultsMarkdown emits", () => { +describe('parseE2Table', () => { + it('round-trips the table that buildResultsMarkdown emits', () => { const parsed = parseE2Table(buildResultsMarkdown(ROWS)); expect(parsed).toHaveLength(ROWS.length); expect(parsed.map((r) => r.n)).toEqual([50, 5000]); @@ -39,7 +39,7 @@ describe("parseE2Table", () => { expect(parsed.map((r) => r.ratioEntries)).toEqual([8.8, 833.8]); }); - it("ignores prose and other tables around the E2 table", () => { + it('ignores prose and other tables around the E2 table', () => { const md = `# Notes | unrelated | table | @@ -50,13 +50,13 @@ ${buildResultsMarkdown(ROWS)}`; expect(parseE2Table(md).map((r) => r.n)).toEqual([50, 5000]); }); - it("throws on a table with no data rows rather than returning nothing", () => { - expect(() => parseE2Table("# Empty\n\nno table here\n")).toThrow(/no E2 table/i); + it('throws on a table with no data rows rather than returning nothing', () => { + expect(() => parseE2Table('# Empty\n\nno table here\n')).toThrow(/no E2 table/i); }); }); -describe("deterministicView", () => { - it("keeps only the environment-independent columns", () => { +describe('deterministicView', () => { + it('keeps only the environment-independent columns', () => { // backendBytes differs between CI and a dev box (+4 bytes, measured), and the ms // columns vary run to run -- so neither can be part of a baseline comparison. expect(deterministicView(ROWS)).toEqual([ @@ -65,12 +65,12 @@ describe("deterministicView", () => { ]); }); - it("is stable across a build/parse round-trip, so a fresh run is comparable", () => { + it('is stable across a build/parse round-trip, so a fresh run is comparable', () => { const reparsed = parseE2Table(buildResultsMarkdown(ROWS)); expect(deterministicView(reparsed)).toEqual(deterministicView(ROWS)); }); - it("is insensitive to byte and timing drift", () => { + it('is insensitive to byte and timing drift', () => { const drifted = ROWS.map((r) => ({ ...r, backendBytes: r.backendBytes + 4, // the CI/local delta @@ -80,15 +80,15 @@ describe("deterministicView", () => { expect(deterministicView(drifted)).toEqual(deterministicView(ROWS)); }); - it("does notice a real change in the entries counts", () => { + it('does notice a real change in the entries counts', () => { const regressed = ROWS.map((r) => ({ ...r, checkpointEntries: r.checkpointEntries + 1 })); expect(deterministicView(regressed)).not.toEqual(deterministicView(ROWS)); }); }); -describe("the committed RESULTS.md baseline", () => { - it("parses, and its deterministic view survives a round-trip", () => { - const md = readFileSync(fileURLToPath(new URL("../RESULTS.md", import.meta.url)), "utf8"); +describe('the committed RESULTS.md baseline', () => { + it('parses, and its deterministic view survives a round-trip', () => { + const md = readFileSync(fileURLToPath(new URL('../RESULTS.md', import.meta.url)), 'utf8'); const baseline = parseE2Table(md); expect(baseline.length).toBeGreaterThan(0); // Guards against a hand-edit that breaks the table shape the E2 gate reads. diff --git a/experiments/test/session-fixture.test.ts b/experiments/test/session-fixture.test.ts index 4837096..0a554fe 100644 --- a/experiments/test/session-fixture.test.ts +++ b/experiments/test/session-fixture.test.ts @@ -1,10 +1,10 @@ -import { describe, it, expect, afterAll } from "vitest"; -import { SessionManager, type FileEntry } from "@earendil-works/pi-coding-agent"; -import { RedisSessionBackend } from "@sh/session-backend"; -import { BufferedRedisBackend } from "@sh/harness/buffered-redis-backend"; -import { buildCompactedSession } from "../src/session-fixture"; +import { describe, it, expect, afterAll } from 'vitest'; +import { SessionManager, type FileEntry } from '@earendil-works/pi-coding-agent'; +import { RedisSessionBackend } from '@sh/session-backend'; +import { BufferedRedisBackend } from '@sh/harness/buffered-redis-backend'; +import { buildCompactedSession } from '../src/session-fixture'; -const REDIS = process.env.REDIS_URL ?? "redis://127.0.0.1:6379"; +const REDIS = process.env.REDIS_URL ?? 'redis://127.0.0.1:6379'; const store = new RedisSessionBackend(REDIS); const sids: string[] = []; @@ -13,8 +13,8 @@ afterAll(async () => { await store.close(); }); -describe("buildCompactedSession", () => { - it("produces a compacted, checkpointed session whose tail is far smaller than the full log", async () => { +describe('buildCompactedSession', () => { + it('produces a compacted, checkpointed session whose tail is far smaller than the full log', async () => { const fx = await buildCompactedSession(store, { n: 40, tailKept: 4 }); sids.push(fx.sessionId); @@ -24,15 +24,19 @@ describe("buildCompactedSession", () => { const backend = new BufferedRedisBackend(store); const marker = await backend.latestCheckpoint(fx.sessionId); - expect((marker as { customType?: string } | null)?.customType).toBe("checkpoint"); + expect((marker as { customType?: string } | null)?.customType).toBe('checkpoint'); }); - it("reconstructs identically via openFromCheckpoint and openFromBackend (parity)", async () => { + it('reconstructs identically via openFromCheckpoint and openFromBackend (parity)', async () => { const fx = await buildCompactedSession(store, { n: 30, tailKept: 4 }); sids.push(fx.sessionId); const backend = new BufferedRedisBackend(store); - const viaCheckpoint = await SessionManager.openFromCheckpoint(fx.sessionId, backend, process.cwd()); + const viaCheckpoint = await SessionManager.openFromCheckpoint( + fx.sessionId, + backend, + process.cwd(), + ); const viaBackend = await SessionManager.openFromBackend(fx.sessionId, backend, process.cwd()); expect(viaCheckpoint.buildSessionContext()).toEqual(viaBackend.buildSessionContext()); }); diff --git a/experiments/test/sharing-benefit.test.ts b/experiments/test/sharing-benefit.test.ts index e1a08c8..986d23d 100644 --- a/experiments/test/sharing-benefit.test.ts +++ b/experiments/test/sharing-benefit.test.ts @@ -1,16 +1,40 @@ -import { describe, it, expect } from "vitest"; -import { reservationBenefit } from "../src/sharing.js"; -describe("reservationBenefit", () => { - it("computes dedicated:shared reservation-seconds ratio and the latency guardrail", () => { - const ded = { arm: "dedicated" as const, resvSecPerLeaf: 120, p95Ms: 10000, throughput: 0.3, peakPods: 8 }; - const shr = { arm: "shared" as const, resvSecPerLeaf: 20, p95Ms: 11000, throughput: 0.29, peakPods: 2 }; +import { describe, it, expect } from 'vitest'; +import { reservationBenefit } from '../src/sharing.js'; +describe('reservationBenefit', () => { + it('computes dedicated:shared reservation-seconds ratio and the latency guardrail', () => { + const ded = { + arm: 'dedicated' as const, + resvSecPerLeaf: 120, + p95Ms: 10000, + throughput: 0.3, + peakPods: 8, + }; + const shr = { + arm: 'shared' as const, + resvSecPerLeaf: 20, + p95Ms: 11000, + throughput: 0.29, + peakPods: 2, + }; const r = reservationBenefit(ded, shr, 2); - expect(r.ratio).toBe(6); // 120/20 + expect(r.ratio).toBe(6); // 120/20 expect(r.withinDegrade).toBe(true); // 11000 <= 2*10000 }); - it("flags degraded latency", () => { - const ded = { arm: "dedicated" as const, resvSecPerLeaf: 120, p95Ms: 5000, throughput: 0.3, peakPods: 8 }; - const shr = { arm: "shared" as const, resvSecPerLeaf: 20, p95Ms: 20000, throughput: 0.1, peakPods: 2 }; + it('flags degraded latency', () => { + const ded = { + arm: 'dedicated' as const, + resvSecPerLeaf: 120, + p95Ms: 5000, + throughput: 0.3, + peakPods: 8, + }; + const shr = { + arm: 'shared' as const, + resvSecPerLeaf: 20, + p95Ms: 20000, + throughput: 0.1, + peakPods: 2, + }; expect(reservationBenefit(ded, shr, 2).withinDegrade).toBe(false); }); }); diff --git a/experiments/test/swebench-deck.test.ts b/experiments/test/swebench-deck.test.ts index 29e32d2..5caee03 100644 --- a/experiments/test/swebench-deck.test.ts +++ b/experiments/test/swebench-deck.test.ts @@ -1,16 +1,16 @@ -import { describe, it, expect } from "vitest"; -import { readFileSync } from "node:fs"; -import { fileURLToPath } from "node:url"; +import { describe, it, expect } from 'vitest'; +import { readFileSync } from 'node:fs'; +import { fileURLToPath } from 'node:url'; const load = (rel: string) => - JSON.parse(readFileSync(fileURLToPath(new URL(rel, import.meta.url)), "utf8")); + JSON.parse(readFileSync(fileURLToPath(new URL(rel, import.meta.url)), 'utf8')); -describe("swebench deck", () => { - const deck = load("../swebench/deck.json"); - const bake = load("../swebench/bake-list.json"); +describe('swebench deck', () => { + const deck = load('../swebench/deck.json'); + const bake = load('../swebench/bake-list.json'); - it("has a stable hash shared by deck and bake-list", () => { - expect(typeof deck.deckHash).toBe("string"); + it('has a stable hash shared by deck and bake-list', () => { + expect(typeof deck.deckHash).toBe('string'); expect(deck.deckHash.length).toBeGreaterThan(0); expect(bake.deckHash).toBe(deck.deckHash); }); @@ -31,46 +31,76 @@ describe("swebench deck", () => { } }); - it("the bake-list carries no repo or env_key absent from the deck (reverse drift guard)", () => { + it('the bake-list carries no repo or env_key absent from the deck (reverse drift guard)', () => { const deckRepos = new Set(deck.instances.map((i: any) => i.repo)); const deckEnvKeys = new Set(deck.instances.map((i: any) => i.env_key)); for (const repo of bake.repos) { - expect(deckRepos.has(repo), `bake-list repo ${repo} not present in the deck (stale entry)`).toBe(true); + expect( + deckRepos.has(repo), + `bake-list repo ${repo} not present in the deck (stale entry)`, + ).toBe(true); } for (const key of bake.envKeys) { - expect(deckEnvKeys.has(key), `bake-list env_key ${key} not present in the deck (stale entry)`).toBe(true); + expect( + deckEnvKeys.has(key), + `bake-list env_key ${key} not present in the deck (stale entry)`, + ).toBe(true); } for (const e of bake.envs) { - expect(deckEnvKeys.has(e.env_key), `bake-list.envs env_key ${e.env_key} not present in the deck (stale entry)`).toBe(true); + expect( + deckEnvKeys.has(e.env_key), + `bake-list.envs env_key ${e.env_key} not present in the deck (stale entry)`, + ).toBe(true); } }); - it("each instance carries the required normalized fields", () => { + it('each instance carries the required normalized fields', () => { for (const inst of deck.instances) { - for (const f of ["instance_id", "repo", "base_commit", "environment_setup_commit", "version", "env_key", "problem_statement"]) { + for (const f of [ + 'instance_id', + 'repo', + 'base_commit', + 'environment_setup_commit', + 'version', + 'env_key', + 'problem_statement', + ]) { expect(inst[f], `${inst.instance_id} missing ${f}`).toBeTruthy(); } expect(Array.isArray(inst.fail_to_pass)).toBe(true); expect(Array.isArray(inst.pass_to_pass)).toBe(true); // runtime/bucket may be null (pre-measurement) or populated (post Task 5) - expect(["light", "medium", "heavy", null]).toContain(inst.weight_bucket); + expect(['light', 'medium', 'heavy', null]).toContain(inst.weight_bucket); } }); - it("each instance carries a canonical gold-test command and directives (Task 5-gen)", () => { + it('each instance carries a canonical gold-test command and directives (Task 5-gen)', () => { for (const inst of deck.instances) { - expect(typeof inst.test_cmd, `${inst.instance_id} test_cmd should be a string`).toBe("string"); - expect(inst.test_cmd.length, `${inst.instance_id} test_cmd should be non-empty`).toBeGreaterThan(0); + expect(typeof inst.test_cmd, `${inst.instance_id} test_cmd should be a string`).toBe( + 'string', + ); + expect( + inst.test_cmd.length, + `${inst.instance_id} test_cmd should be non-empty`, + ).toBeGreaterThan(0); - expect(Array.isArray(inst.test_directives), `${inst.instance_id} test_directives should be an array`).toBe(true); - expect(inst.test_directives.length, `${inst.instance_id} test_directives should be non-empty`).toBeGreaterThan(0); + expect( + Array.isArray(inst.test_directives), + `${inst.instance_id} test_directives should be an array`, + ).toBe(true); + expect( + inst.test_directives.length, + `${inst.instance_id} test_directives should be non-empty`, + ).toBeGreaterThan(0); for (const d of inst.test_directives) { - expect(typeof d, `${inst.instance_id} test_directives entries should be strings`).toBe("string"); + expect(typeof d, `${inst.instance_id} test_directives entries should be strings`).toBe( + 'string', + ); } } }); - it("every deck env_key has a bake-list.envs entry whose representative_instance_id is a deck instance for that env_key", () => { + it('every deck env_key has a bake-list.envs entry whose representative_instance_id is a deck instance for that env_key', () => { expect(Array.isArray(bake.envs)).toBe(true); const instancesByEnvKey = new Map>(); @@ -83,7 +113,7 @@ describe("swebench deck", () => { const envsByKey = new Map(); for (const e of bake.envs) { - for (const f of ["env_key", "repo", "representative_instance_id", "instance_image_key"]) { + for (const f of ['env_key', 'repo', 'representative_instance_id', 'instance_image_key']) { expect(e[f], `bake-list env entry missing ${f}: ${JSON.stringify(e)}`).toBeTruthy(); } envsByKey.set(e.env_key, e); @@ -105,7 +135,7 @@ describe("swebench deck", () => { ).toBe(repoByInstance.get(entry.representative_instance_id)); // instance_image_key must embed the representative_instance_id; swebench maps "__" -> "_1776_" // in image tags (e.g. django__django-11555 -> ...django_1776_django-11555...). - const imageId = entry.representative_instance_id.replace(/__/g, "_1776_"); + const imageId = entry.representative_instance_id.replace(/__/g, '_1776_'); expect( entry.instance_image_key.includes(imageId), `instance_image_key ${entry.instance_image_key} does not embed representative id ${imageId}`, diff --git a/experiments/test/swebench-sandbox-build.test.ts b/experiments/test/swebench-sandbox-build.test.ts index b53e535..901dd9d 100644 --- a/experiments/test/swebench-sandbox-build.test.ts +++ b/experiments/test/swebench-sandbox-build.test.ts @@ -5,25 +5,25 @@ // Asserts against the emitter's REAL stdout (not a hand-maintained copy): // invokes the bash emitter via execSync and parses the emitted Dockerfile // text. Reads only the committed bake-list.json; no network, docker, or oc. -import { describe, it, expect, beforeAll } from "vitest"; -import { execSync } from "node:child_process"; -import { readFileSync } from "node:fs"; -import { fileURLToPath } from "node:url"; +import { describe, it, expect, beforeAll } from 'vitest'; +import { execSync } from 'node:child_process'; +import { readFileSync } from 'node:fs'; +import { fileURLToPath } from 'node:url'; const load = (rel: string) => - JSON.parse(readFileSync(fileURLToPath(new URL(rel, import.meta.url)), "utf8")); + JSON.parse(readFileSync(fileURLToPath(new URL(rel, import.meta.url)), 'utf8')); // experiments/test/ -> repo root is two levels up. -const repoRoot = fileURLToPath(new URL("../..", import.meta.url)); +const repoRoot = fileURLToPath(new URL('../..', import.meta.url)); // Mirrors the env_dir() sanitizer documented in build-swebench-sandbox.sh: // strip the trailing ":" suffix, then replace any remaining "/" or ":" // with "-". Kept here ONLY to compute the expected value independently of // the script under test; the script's own comment is the source of truth. -const envDir = (envKey: string) => envKey.replace(/:[^:]*$/, "").replace(/[/:]/g, "-"); +const envDir = (envKey: string) => envKey.replace(/:[^:]*$/, '').replace(/[/:]/g, '-'); -describe("swebench-sandbox Dockerfile emitter (build-swebench-sandbox.sh --emit --limit 3)", () => { - const bake = load("../swebench/bake-list.json"); +describe('swebench-sandbox Dockerfile emitter (build-swebench-sandbox.sh --emit --limit 3)', () => { + const bake = load('../swebench/bake-list.json'); const selected = [...bake.envs] .sort((a: any, b: any) => (a.env_key < b.env_key ? -1 : a.env_key > b.env_key ? 1 : 0)) .slice(0, 3); @@ -31,29 +31,29 @@ describe("swebench-sandbox Dockerfile emitter (build-swebench-sandbox.sh --emit let dockerfile: string; beforeAll(() => { - dockerfile = execSync("bash deploy/knative/build-swebench-sandbox.sh --emit --limit 3", { + dockerfile = execSync('bash deploy/knative/build-swebench-sandbox.sh --emit --limit 3', { cwd: repoRoot, - encoding: "utf8", + encoding: 'utf8', }); }); - it("selects exactly 3 distinct env-keys from the committed bake-list, sorted", () => { + it('selects exactly 3 distinct env-keys from the committed bake-list, sorted', () => { expect(selected).toHaveLength(3); expect(new Set(selected.map((e: any) => e.env_key)).size).toBe(3); }); - it("--print-tag derives -of from the bake-list for each --limit", () => { + it('--print-tag derives -of from the bake-list for each --limit', () => { const printTag = (limit: number) => execSync(`bash deploy/knative/build-swebench-sandbox.sh --print-tag --limit ${limit}`, { cwd: repoRoot, - encoding: "utf8", + encoding: 'utf8', }).trim(); const total = bake.envs.length; expect(printTag(3)).toBe(`${bake.deckHash}-3of${total}`); expect(printTag(total)).toBe(`${bake.deckHash}-${total}of${total}`); }); - it("emits exactly 3 env_N build stages, each FROM the correct instance_image_key", () => { + it('emits exactly 3 env_N build stages, each FROM the correct instance_image_key', () => { const stageMatches = [...dockerfile.matchAll(/^FROM (\S+) AS env_(\d+)$/gm)]; expect(stageMatches).toHaveLength(3); const byIndex = new Map(stageMatches.map((m) => [Number(m[2]), m[1]])); @@ -67,19 +67,15 @@ describe("swebench-sandbox Dockerfile emitter (build-swebench-sandbox.sh --emit // envs (matplotlib/sklearn numpy import failed) per the Task-3b verify gate. // 'conda create --clone' copies all files faithfully with self-consistent // prefixes at the same /opt/miniconda3 base. - const cloneCmds = [ - ...dockerfile.matchAll(/conda create --clone testbed -n \S+ -y/g), - ]; + const cloneCmds = [...dockerfile.matchAll(/conda create --clone testbed -n \S+ -y/g)]; expect(cloneCmds).toHaveLength(3); // one per env stage for (const env of selected) { const dir = envDir(env.env_key); - expect(dockerfile).toContain( - `/opt/miniconda3/bin/conda create --clone testbed -n ${dir} -y`, - ); + expect(dockerfile).toContain(`/opt/miniconda3/bin/conda create --clone testbed -n ${dir} -y`); } }); - it("COPYs each cloned env to the exact same /opt/miniconda3/envs/ path (no relocation)", () => { + it('COPYs each cloned env to the exact same /opt/miniconda3/envs/ path (no relocation)', () => { // conda clone already wrote correct prefixes for this path, so source==dest // and the env is usable as-is (activatable via // 'source /opt/miniconda3/envs//bin/activate'). @@ -94,48 +90,50 @@ describe("swebench-sandbox Dockerfile emitter (build-swebench-sandbox.sh --emit expect(dirs.size).toBe(3); }); - it("git config --system --add safe.directory is present (root-cloned repos, non-root pod)", () => { + it('git config --system --add safe.directory is present (root-cloned repos, non-root pod)', () => { expect(dockerfile).toContain("git config --system --add safe.directory '*'"); }); - it("contains NONE of the abandoned conda-pack machinery (regression guard)", () => { + it('contains NONE of the abandoned conda-pack machinery (regression guard)', () => { for (const banned of [ - "conda pack", - "conda-unpack", - "--ignore-editable-packages", - "--ignore-missing-files", - "tar -xzf", + 'conda pack', + 'conda-unpack', + '--ignore-editable-packages', + '--ignore-missing-files', + 'tar -xzf', ]) { expect(dockerfile).not.toContain(banned); } }); - it("mirrors every unique slice repo with git clone --mirror", () => { + it('mirrors every unique slice repo with git clone --mirror', () => { const repos = [...new Set(selected.map((e: any) => e.repo))]; expect(repos.length).toBeGreaterThan(0); for (const repo of repos) { - expect(dockerfile).toContain(`git clone --mirror https://github.com/${repo}.git /repos/${repo}.git`); + expect(dockerfile).toContain( + `git clone --mirror https://github.com/${repo}.git /repos/${repo}.git`, + ); } }); - it("carries both deck labels with correct values", () => { + it('carries both deck labels with correct values', () => { expect(dockerfile).toContain(`LABEL sh.kagenti.io/deck-hash="${bake.deckHash}"`); expect(dockerfile).toContain(`LABEL sh.kagenti.io/deck-slice="3of${bake.envs.length}"`); }); - it("runs as non-root 65532 and its terminal CMD is sleep infinity", () => { - expect(dockerfile).toContain("USER 65532"); + it('runs as non-root 65532 and its terminal CMD is sleep infinity', () => { + expect(dockerfile).toContain('USER 65532'); expect(dockerfile.trim().endsWith('CMD ["sleep","infinity"]')).toBe(true); }); - it("is pure/offline: mentions no docker/oc invocation in the emitted text", () => { + it('is pure/offline: mentions no docker/oc invocation in the emitted text', () => { expect(dockerfile).not.toMatch(/\boc start-build\b/); expect(dockerfile).not.toMatch(/\bdocker build\b/); }); }); -describe("swebench-sandbox emitter — iterative accumulation (--offset / --base / base-tools)", () => { - const bake = load("../swebench/bake-list.json"); +describe('swebench-sandbox emitter — iterative accumulation (--offset / --base / base-tools)', () => { + const bake = load('../swebench/bake-list.json'); const sorted = [...bake.envs].sort((a: any, b: any) => a.env_key < b.env_key ? -1 : a.env_key > b.env_key ? 1 : 0, ); @@ -144,16 +142,16 @@ describe("swebench-sandbox emitter — iterative accumulation (--offset / --base const emit = (args: string) => execSync(`bash deploy/knative/build-swebench-sandbox.sh --emit ${args}`, { cwd: repoRoot, - encoding: "utf8", + encoding: 'utf8', }); const printTag = (args: string) => execSync(`bash deploy/knative/build-swebench-sandbox.sh --print-tag ${args}`, { cwd: repoRoot, - encoding: "utf8", + encoding: 'utf8', }).trim(); - it("--offset 5 --limit 5 selects envs[5:10] (correct instance_image_keys)", () => { - const df = emit("--offset 5 --limit 5 --base prior"); + it('--offset 5 --limit 5 selects envs[5:10] (correct instance_image_keys)', () => { + const df = emit('--offset 5 --limit 5 --base prior'); const stages = [...df.matchAll(/^FROM (\S+) AS env_(\d+)$/gm)]; expect(stages).toHaveLength(5); const byIndex = new Map(stages.map((m) => [Number(m[2]), m[1]])); @@ -162,40 +160,40 @@ describe("swebench-sandbox emitter — iterative accumulation (--offset / --base }); }); - it("--base sets the assembled FROM image", () => { - expect(emit("--offset 5 --limit 5 --base my/prior:img")).toContain( - "FROM my/prior:img AS assembled", + it('--base sets the assembled FROM image', () => { + expect(emit('--offset 5 --limit 5 --base my/prior:img')).toContain( + 'FROM my/prior:img AS assembled', ); }); - it("batch-2 style (base set, no base tools) OMITS apt + safe.directory", () => { - const df = emit("--offset 5 --limit 5 --base prior"); - expect(df).not.toContain("apt-get install"); - expect(df).not.toContain("safe.directory"); + it('batch-2 style (base set, no base tools) OMITS apt + safe.directory', () => { + const df = emit('--offset 5 --limit 5 --base prior'); + expect(df).not.toContain('apt-get install'); + expect(df).not.toContain('safe.directory'); // still switches to root for the COPY/RUN steps and back to 65532 at the end - expect(df).toContain("FROM prior AS assembled\nUSER 0"); - expect(df).toContain("USER 65532"); + expect(df).toContain('FROM prior AS assembled\nUSER 0'); + expect(df).toContain('USER 65532'); }); - it("batch-1 default base still emits apt + safe.directory (auto base-tools)", () => { - const df = emit("--offset 0 --limit 5"); - expect(df).toContain("FROM ubuntu:22.04 AS assembled"); + it('batch-1 default base still emits apt + safe.directory (auto base-tools)', () => { + const df = emit('--offset 0 --limit 5'); + expect(df).toContain('FROM ubuntu:22.04 AS assembled'); expect(df).toContain( - "apt-get install -y --no-install-recommends git ripgrep ca-certificates " + - "build-essential python3-dev pkg-config libfreetype6-dev libpng-dev", + 'apt-get install -y --no-install-recommends git ripgrep ca-certificates ' + + 'build-essential python3-dev pkg-config libfreetype6-dev libpng-dev', ); expect(df).toContain("git config --system --add safe.directory '*'"); }); - it("--no-base-tools force-skips tooling even on the default ubuntu base", () => { - const df = emit("--offset 0 --limit 5 --no-base-tools"); - expect(df).toContain("FROM ubuntu:22.04 AS assembled"); - expect(df).not.toContain("apt-get install"); - expect(df).not.toContain("safe.directory"); + it('--no-base-tools force-skips tooling even on the default ubuntu base', () => { + const df = emit('--offset 0 --limit 5 --no-base-tools'); + expect(df).toContain('FROM ubuntu:22.04 AS assembled'); + expect(df).not.toContain('apt-get install'); + expect(df).not.toContain('safe.directory'); }); it("repo clones use the idempotent 'test -d ... ||' form", () => { - const df = emit("--offset 5 --limit 5 --base prior"); + const df = emit('--offset 5 --limit 5 --base prior'); const uniqueRepos = [...new Set(sorted.slice(5, 10).map((e: any) => e.repo))]; expect(uniqueRepos.length).toBeGreaterThan(0); for (const repo of uniqueRepos) { @@ -205,18 +203,18 @@ describe("swebench-sandbox emitter — iterative accumulation (--offset / --base } }); - it("deck-slice label is CUMULATIVE (offset + selected) coverage", () => { - expect(emit("--offset 5 --limit 5 --base prior")).toContain( + it('deck-slice label is CUMULATIVE (offset + selected) coverage', () => { + expect(emit('--offset 5 --limit 5 --base prior')).toContain( `LABEL sh.kagenti.io/deck-slice="10of${total}"`, ); - expect(emit("--offset 10 --limit 5 --base prior")).toContain( + expect(emit('--offset 10 --limit 5 --base prior')).toContain( `LABEL sh.kagenti.io/deck-slice="${total}of${total}"`, ); }); - it("--print-tag honors --offset (cumulative)", () => { - expect(printTag("--offset 0 --limit 5")).toBe(`${bake.deckHash}-5of${total}`); - expect(printTag("--offset 5 --limit 5")).toBe(`${bake.deckHash}-10of${total}`); - expect(printTag("--offset 10 --limit 5")).toBe(`${bake.deckHash}-${total}of${total}`); + it('--print-tag honors --offset (cumulative)', () => { + expect(printTag('--offset 0 --limit 5')).toBe(`${bake.deckHash}-5of${total}`); + expect(printTag('--offset 5 --limit 5')).toBe(`${bake.deckHash}-10of${total}`); + expect(printTag('--offset 10 --limit 5')).toBe(`${bake.deckHash}-${total}of${total}`); }); }); diff --git a/experiments/test/workload.test.ts b/experiments/test/workload.test.ts index 2e4afe3..7b262ab 100644 --- a/experiments/test/workload.test.ts +++ b/experiments/test/workload.test.ts @@ -1,48 +1,50 @@ -import { describe, it, expect } from "vitest"; -import { fileURLToPath } from "node:url"; -import { getWorkloadProvider } from "../src/workload.js"; +import { describe, it, expect } from 'vitest'; +import { fileURLToPath } from 'node:url'; +import { getWorkloadProvider } from '../src/workload.js'; -const deckPath = fileURLToPath(new URL("../swebench/deck.json", import.meta.url)); +const deckPath = fileURLToPath(new URL('../swebench/deck.json', import.meta.url)); -describe("synthetic provider", () => { - const p = getWorkloadProvider({ WORKLOAD: "synthetic" }); - it("is the default when WORKLOAD is unset", () => { - expect(getWorkloadProvider({}).name).toBe("synthetic"); +describe('synthetic provider', () => { + const p = getWorkloadProvider({ WORKLOAD: 'synthetic' }); + it('is the default when WORKLOAD is unset', () => { + expect(getWorkloadProvider({}).name).toBe('synthetic'); }); - it("yields L0/L1/L2 code-review items", () => { + it('yields L0/L1/L2 code-review items', () => { const items = p.curveItems(); - expect(items.map((i) => i.label)).toEqual(["L0", "L1", "L2"]); - expect(items[0].post).toEqual({ item: { item_id: "L0", file: "small.py", pattern: "password" } }); - expect(p.sweepItem().label).toBe("L2"); + expect(items.map((i) => i.label)).toEqual(['L0', 'L1', 'L2']); + expect(items[0].post).toEqual({ + item: { item_id: 'L0', file: 'small.py', pattern: 'password' }, + }); + expect(p.sweepItem().label).toBe('L2'); }); }); -describe("swebench provider", () => { - const p = getWorkloadProvider({ WORKLOAD: "swebench" }, deckPath); - it("curveItems has exactly the three buckets, deterministic representatives", () => { +describe('swebench provider', () => { + const p = getWorkloadProvider({ WORKLOAD: 'swebench' }, deckPath); + it('curveItems has exactly the three buckets, deterministic representatives', () => { const items = p.curveItems(); - expect(items.map((i) => i.label).sort()).toEqual(["heavy", "light", "medium"]); - const again = getWorkloadProvider({ WORKLOAD: "swebench" }, deckPath).curveItems(); + expect(items.map((i) => i.label).sort()).toEqual(['heavy', 'light', 'medium']); + const again = getWorkloadProvider({ WORKLOAD: 'swebench' }, deckPath).curveItems(); expect(items.map((i) => i.instanceId)).toEqual(again.map((i) => i.instanceId)); // stable }); - it("emits a well-formed solve envelope per item", () => { - const it0 = p.curveItems().find((i) => i.label === "heavy")!; - expect(it0.post.kind).toBe("solve"); - expect(typeof it0.post.problemStatement).toBe("string"); + it('emits a well-formed solve envelope per item', () => { + const it0 = p.curveItems().find((i) => i.label === 'heavy')!; + expect(it0.post.kind).toBe('solve'); + expect(typeof it0.post.problemStatement).toBe('string'); expect((it0.post.problemStatement as string).length).toBeGreaterThan(0); - expect(it0.post.ref).toMatch(/^[0-9a-f]{7,40}$/); // base_commit - expect(it0.post.repoUrl).toMatch(/^\/repos\/.+\.git$/); // baked bare mirror path - expect(it0.post.env_key).toMatch(/:latest$/); // passed through verbatim + expect(it0.post.ref).toMatch(/^[0-9a-f]{7,40}$/); // base_commit + expect(it0.post.repoUrl).toMatch(/^\/repos\/.+\.git$/); // baked bare mirror path + expect(it0.post.env_key).toMatch(/:latest$/); // passed through verbatim }); - it("sweepItem is a heavy instance", () => { - expect(p.sweepItem().label).toBe("heavy"); + it('sweepItem is a heavy instance', () => { + expect(p.sweepItem().label).toBe('heavy'); }); - it("sliceItems is deterministic and bucket-balanced", () => { + it('sliceItems is deterministic and bucket-balanced', () => { const a = p.sliceItems({ perBucket: 2, seed: 7 }); const b = p.sliceItems({ perBucket: 2, seed: 7 }); expect(a.map((i) => i.instanceId)).toEqual(b.map((i) => i.instanceId)); const c = p.sliceItems({ perBucket: 2, seed: 99 }); expect(a.map((i) => i.instanceId)).not.toEqual(c.map((i) => i.instanceId)); // seed changes selection - expect(a.filter((i) => i.label === "light").length).toBe(2); + expect(a.filter((i) => i.label === 'light').length).toBe(2); }); }); diff --git a/experiments/vitest.config.ts b/experiments/vitest.config.ts index c794660..e1e28f8 100644 --- a/experiments/vitest.config.ts +++ b/experiments/vitest.config.ts @@ -1,5 +1,5 @@ -import { defineConfig } from "vitest/config"; +import { defineConfig } from 'vitest/config'; export default defineConfig({ - test: { include: ["test/**/*.test.ts"] }, + test: { include: ['test/**/*.test.ts'] }, }); diff --git a/harness/README.md b/harness/README.md index 54d7708..95e0e92 100644 --- a/harness/README.md +++ b/harness/README.md @@ -4,12 +4,14 @@ Serverless-harness glue: adapts the generic `@sh/session-backend` log store to P `SessionStorageBackend`, with write-behind durability. ## Components + - `BufferedRedisBackend` — write-behind decorator (queue + `flush()`); implements Pi's `SessionStorageBackend` over a `LogStore` (`RedisSessionBackend`). - `flushExtension` — flushes at `turn_end` and `session_shutdown`. - `cli.ts` — headless one-shot smoke entry (resume via `PI_SESSION_ID`). ## Prerequisites + The `pi-fork` workspace packages must be built before the harness can import the compiled `@earendil-works/pi-coding-agent` (and `@earendil-works/pi-ai`). Build them in dependency order: @@ -19,10 +21,12 @@ for p in ai agent tui coding-agent; do pnpm -C pi-fork/packages/$p build; done ``` ## Tests + - `pnpm -C harness test` — decorator units + the SessionManager↔Redis integration test (needs Redis at `REDIS_URL`, default `redis://127.0.0.1:6379`). ## Headless smoke + - `pnpm -C harness exec tsx src/cli.ts ""` runs one turn; set `PI_SESSION_ID` to resume an existing session from Redis. Requires a model credential. - Gateway: pinned Pi reads `ANTHROPIC_API_KEY` and a fixed base URL. `cli.ts` includes an @@ -30,5 +34,6 @@ for p in ai agent tui coding-agent; do pnpm -C pi-fork/packages/$p build; done `ANTHROPIC_AUTH_TOKEN` (base URL is overridden and auth is sent as `Authorization: Bearer`). ## Dependency direction + `harness → { pi-fork, @sh/session-backend }`. Pi core never imports Redis or `@sh/session-backend`. diff --git a/harness/src/budget-voter.ts b/harness/src/budget-voter.ts index 10bc868..41607d0 100644 --- a/harness/src/budget-voter.ts +++ b/harness/src/budget-voter.ts @@ -1,4 +1,8 @@ -import type { ExtensionContext, ExtensionFactory, SessionManager } from "@earendil-works/pi-coding-agent"; +import type { + ExtensionContext, + ExtensionFactory, + SessionManager, +} from '@earendil-works/pi-coding-agent'; export interface BudgetState { spent: number; @@ -6,15 +10,14 @@ export interface BudgetState { limit: number; } export type BudgetDecision = - | { decision: "commit" } - | { decision: "abort"; reason: "budget_exceeded" }; + { decision: 'commit' } | { decision: 'abort'; reason: 'budget_exceeded' }; /** Pure policy. Disabled (always commits) when limit is non-finite or <= 0. */ export function decideBudget(s: BudgetState): BudgetDecision { - if (!Number.isFinite(s.limit) || s.limit <= 0) return { decision: "commit" }; + if (!Number.isFinite(s.limit) || s.limit <= 0) return { decision: 'commit' }; return s.spent + s.estimated > s.limit - ? { decision: "abort", reason: "budget_exceeded" } - : { decision: "commit" }; + ? { decision: 'abort', reason: 'budget_exceeded' } + : { decision: 'commit' }; } /** @@ -24,13 +27,16 @@ export function decideBudget(s: BudgetState): BudgetDecision { * loaded SessionManager — the voter does not depend on a session_start event (see below). */ export function branchSpend(sm: { getBranch?: () => unknown[] } | undefined): number | null { - if (!sm || typeof sm.getBranch !== "function") return null; + if (!sm || typeof sm.getBranch !== 'function') return null; let total = 0; for (const entry of sm.getBranch() as Array<{ type?: string; - message?: { role?: string; usage?: { input: number; output: number; cacheRead: number; cacheWrite: number } }; + message?: { + role?: string; + usage?: { input: number; output: number; cacheRead: number; cacheWrite: number }; + }; }>) { - if (entry?.type === "message" && entry.message?.role === "assistant" && entry.message.usage) { + if (entry?.type === 'message' && entry.message?.role === 'assistant' && entry.message.usage) { const u = entry.message.usage; total += u.input + u.output + u.cacheRead + u.cacheWrite; } @@ -60,14 +66,14 @@ export function budgetVoterExtension( ): ExtensionFactory { const baseline = opts.baseline ?? 0; return (pi) => { - pi.on("tool_call", (_e, ctx) => { + pi.on('tool_call', (_e, ctx) => { const total = sessionSpendTotal(ctx); if (total == null) return {}; // defensive: don't block when spend is unknown const spent = total - baseline; const d = decideBudget({ spent, estimated: opts.margin ?? 0, limit: opts.limit }); - if (d.decision === "abort") { - sm.appendCustomEntry("abort", { reason: d.reason, spent, limit: opts.limit }); - return { block: true, reason: "Session token budget exceeded" }; + if (d.decision === 'abort') { + sm.appendCustomEntry('abort', { reason: d.reason, spent, limit: opts.limit }); + return { block: true, reason: 'Session token budget exceeded' }; } return {}; }); diff --git a/harness/src/buffered-redis-backend.ts b/harness/src/buffered-redis-backend.ts index 88f288a..cea55cf 100644 --- a/harness/src/buffered-redis-backend.ts +++ b/harness/src/buffered-redis-backend.ts @@ -1,5 +1,5 @@ -import type { FileEntry, SessionStorageBackend } from "@earendil-works/pi-coding-agent"; -import type { LogStore } from "@sh/session-backend"; +import type { FileEntry, SessionStorageBackend } from '@earendil-works/pi-coding-agent'; +import type { LogStore } from '@sh/session-backend'; /** * Write-behind decorator adapting a generic LogStore to Pi's SessionStorageBackend. @@ -42,7 +42,7 @@ export class BufferedRedisBackend implements SessionStorageBackend { async latestCheckpoint(sessionId: string): Promise { const row = await this.store.latestWhere( sessionId, - (e) => e.type === "custom" && (e as { customType?: string }).customType === "checkpoint", + (e) => e.type === 'custom' && (e as { customType?: string }).customType === 'checkpoint', ); return row ? row.entry : null; } diff --git a/harness/src/checkpoint-extension.ts b/harness/src/checkpoint-extension.ts index d5333a7..17f696d 100644 --- a/harness/src/checkpoint-extension.ts +++ b/harness/src/checkpoint-extension.ts @@ -1,5 +1,5 @@ -import type { ExtensionFactory, FileEntry, SessionManager } from "@earendil-works/pi-coding-agent"; -import type { LogStore } from "@sh/session-backend"; +import type { ExtensionFactory, FileEntry, SessionManager } from '@earendil-works/pi-coding-agent'; +import type { LogStore } from '@sh/session-backend'; /** * On each native compaction, append a tiny resume-pointer marker recording the log @@ -7,15 +7,18 @@ import type { LogStore } from "@sh/session-backend"; * SessionManager (not the store directly) so it flows through the buffered backend * and the existing flush path, keeping positions consistent with Pi's append order. */ -export function checkpointExtension(store: LogStore, sm: SessionManager): ExtensionFactory { +export function checkpointExtension( + store: LogStore, + sm: SessionManager, +): ExtensionFactory { return (pi) => { - pi.on("session_compact", async (e) => { + pi.on('session_compact', async (e) => { const firstKeptEntryId = (e as { compactionEntry?: { firstKeptEntryId?: string } }) .compactionEntry?.firstKeptEntryId; if (!firstKeptEntryId) return; const pos = await store.positionOfId(sm.getSessionId(), firstKeptEntryId); if (pos != null) { - sm.appendCustomEntry("checkpoint", { resumeFromPosition: pos }); + sm.appendCustomEntry('checkpoint', { resumeFromPosition: pos }); } }); }; diff --git a/harness/src/classify-outcome.ts b/harness/src/classify-outcome.ts index e977867..7de2977 100644 --- a/harness/src/classify-outcome.ts +++ b/harness/src/classify-outcome.ts @@ -1,4 +1,4 @@ -import type { LeafResult } from "./run-leaf.js"; +import type { LeafResult } from './run-leaf.js'; export interface Outcome { ack: boolean; @@ -13,7 +13,7 @@ export interface Outcome { * drains as pool leases free (spec §4.3). (A process crash never returns here → stays pending → reclaimed.) */ export function classifyOutcome(result: LeafResult): Outcome { - if (result.status === "failed" && (result.reason === "error" || result.reason === "saturated")) { + if (result.status === 'failed' && (result.reason === 'error' || result.reason === 'saturated')) { return { ack: false, retryable: true }; } return { ack: true, retryable: false }; diff --git a/harness/src/cli.ts b/harness/src/cli.ts index 0e3bcc1..e0b058b 100644 --- a/harness/src/cli.ts +++ b/harness/src/cli.ts @@ -1,4 +1,4 @@ -import { runTurn } from "./run-turn.js"; +import { runTurn } from './run-turn.js'; async function main() { const prompt = process.argv[2]; diff --git a/harness/src/converge.ts b/harness/src/converge.ts index 95437dc..0b4be44 100644 --- a/harness/src/converge.ts +++ b/harness/src/converge.ts @@ -1,4 +1,4 @@ -import type { SandboxTransport } from "@sh/k8s-sandbox"; +import type { SandboxTransport } from '@sh/k8s-sandbox'; /** Single-quote-escape a string for safe interpolation into a bash command. */ function sq(s: string): string { @@ -37,7 +37,7 @@ export function buildConvergeScript(repoUrl: string, ref: string, runId: string) `COMMIT=$(git -C "$REPO" rev-parse FETCH_HEAD)`, `[ -d "$LEAF" ] || git -C "$REPO" worktree add --quiet --detach "$LEAF" "$COMMIT"`, `printf '%s' "$LEAF"`, - ].join("\n"); + ].join('\n'); } /** Remove the per-leaf worktree and prune orphans (best-effort; never fails the leaf). */ @@ -48,24 +48,37 @@ export function buildCleanupScript(runId: string): string { `REPO=/workspace/repo; LEAF=${sq(LEAF)}`, `git -C "$REPO" worktree remove --force "$LEAF" 2>/dev/null || rm -rf "$LEAF"`, `git -C "$REPO" worktree prune 2>/dev/null || true`, - ].join("\n"); + ].join('\n'); } /** Run the converge script in the pod; return the worktree ref. Throws on non-zero exit. */ export async function convergeWorkspace( - transport: SandboxTransport, repoUrl: string, ref: string, runId: string, + transport: SandboxTransport, + repoUrl: string, + ref: string, + runId: string, ): Promise { - const { stdout, exitCode, truncated } = await transport.exec(buildConvergeScript(repoUrl, ref, runId), { - timeout: 300, - }); - if (truncated) throw new Error(`converge exceeded the sandbox output cap (converge output too large): ${runId}`); + const { stdout, exitCode, truncated } = await transport.exec( + buildConvergeScript(repoUrl, ref, runId), + { + timeout: 300, + }, + ); + if (truncated) + throw new Error( + `converge exceeded the sandbox output cap (converge output too large): ${runId}`, + ); if (exitCode !== 0) throw new Error(`converge failed (exit ${exitCode})`); return stdout.toString().trim() || leafWorkspaceRef(runId); } /** Best-effort worktree cleanup; swallows errors so it never masks a verdict. */ export async function cleanupWorkspace(transport: SandboxTransport, runId: string): Promise { - try { await transport.exec(buildCleanupScript(runId), { timeout: 60 }); } catch { /* ignore */ } + try { + await transport.exec(buildCleanupScript(runId), { timeout: 60 }); + } catch { + /* ignore */ + } } /** Stage every edit in the leaf worktree and print the resulting unified diff (vs the pinned base). */ @@ -76,18 +89,24 @@ export function buildDiffCaptureScript(runId: string): string { `LEAF=${sq(LEAF)}`, `git -C "$LEAF" add -A`, `git -C "$LEAF" diff --cached`, - ].join("\n"); + ].join('\n'); } /** Run the diff-capture script in the pod; return the patch (possibly empty). Throws on non-zero exit. */ -export async function captureWorkspaceDiff(transport: SandboxTransport, runId: string): Promise { - const { stdout, exitCode, truncated } = await transport.exec(buildDiffCaptureScript(runId), { timeout: 120 }); - if (truncated) throw new Error(`diff capture exceeded the sandbox output cap (diff too large): ${runId}`); +export async function captureWorkspaceDiff( + transport: SandboxTransport, + runId: string, +): Promise { + const { stdout, exitCode, truncated } = await transport.exec(buildDiffCaptureScript(runId), { + timeout: 120, + }); + if (truncated) + throw new Error(`diff capture exceeded the sandbox output cap (diff too large): ${runId}`); if (exitCode !== 0) throw new Error(`diff capture failed (exit ${exitCode})`); const patch = stdout.toString(); // A unified diff must end with a newline. `git diff` emits one, but some exec transports strip // the trailing newline from captured stdout — and a patch that ends mid-line is rejected by // `git apply` / GNU patch ("patch unexpectedly ends in middle of line"), so the captured // model_patch fails to apply during offline evaluation. Restore it for a non-empty patch. - return patch && !patch.endsWith("\n") ? patch + "\n" : patch; + return patch && !patch.endsWith('\n') ? patch + '\n' : patch; } diff --git a/harness/src/flush-extension.ts b/harness/src/flush-extension.ts index a92d33c..0ed28f4 100644 --- a/harness/src/flush-extension.ts +++ b/harness/src/flush-extension.ts @@ -1,5 +1,5 @@ -import type { ExtensionFactory } from "@earendil-works/pi-coding-agent"; -import type { BufferedRedisBackend } from "./buffered-redis-backend.js"; +import type { ExtensionFactory } from '@earendil-works/pi-coding-agent'; +import type { BufferedRedisBackend } from './buffered-redis-backend.js'; /** * Returns a Pi ExtensionFactory that flushes the write-behind buffer at the two @@ -10,9 +10,9 @@ import type { BufferedRedisBackend } from "./buffered-redis-backend.js"; */ export function flushExtension(backend: BufferedRedisBackend): ExtensionFactory { return (pi) => { - pi.on("turn_end", () => backend.flush()); + pi.on('turn_end', () => backend.flush()); // session_shutdown fires on interactive shutdown/reload; it does NOT fire in the headless // runTurn path (which relies on turn_end above + an explicit final backend.flush()). - pi.on("session_shutdown", () => backend.flush()); + pi.on('session_shutdown', () => backend.flush()); }; } diff --git a/harness/src/gate.ts b/harness/src/gate.ts index 5a68479..2833c5e 100644 --- a/harness/src/gate.ts +++ b/harness/src/gate.ts @@ -1,4 +1,4 @@ -export type GateAction = "approve" | "reject" | "abort"; +export type GateAction = 'approve' | 'reject' | 'abort'; export interface GateRequest { gateId: number; @@ -15,42 +15,51 @@ export interface Decision { /** Same shape as Decision; persisted as a durable custom entry to mark a gate consumed. */ export type GateDecision = Decision; -export const GATE_REQUEST_ENTRY_TYPE = "gate-request"; -export const GATE_DECISION_ENTRY_TYPE = "gate-decision"; +export const GATE_REQUEST_ENTRY_TYPE = 'gate-request'; +export const GATE_DECISION_ENTRY_TYPE = 'gate-decision'; export function validateDecision( obj: unknown, ): { ok: true; value: Decision } | { ok: false; error: string } { - if (typeof obj !== "object" || obj === null) return { ok: false, error: "decision must be an object" }; + if (typeof obj !== 'object' || obj === null) + return { ok: false, error: 'decision must be an object' }; const o = obj as Record; - if (typeof o.gateId !== "number" || !Number.isInteger(o.gateId) || o.gateId < 0) { - return { ok: false, error: "gateId must be a non-negative integer" }; + if (typeof o.gateId !== 'number' || !Number.isInteger(o.gateId) || o.gateId < 0) { + return { ok: false, error: 'gateId must be a non-negative integer' }; } - if (o.action !== "approve" && o.action !== "reject" && o.action !== "abort") { + if (o.action !== 'approve' && o.action !== 'reject' && o.action !== 'abort') { return { ok: false, error: 'action must be "approve", "reject", or "abort"' }; } - if (o.feedback !== undefined && typeof o.feedback !== "string") { - return { ok: false, error: "feedback must be a string when present" }; + if (o.feedback !== undefined && typeof o.feedback !== 'string') { + return { ok: false, error: 'feedback must be a string when present' }; } - return { ok: true, value: { gateId: o.gateId, action: o.action, feedback: o.feedback as string | undefined } }; + return { + ok: true, + value: { gateId: o.gateId, action: o.action, feedback: o.feedback as string | undefined }, + }; } type CustomEntry = { type?: string; customType?: string; data?: unknown }; export function isGateRequestEntry(entry: unknown): boolean { const e = entry as CustomEntry | null; - return !!e && e.type === "custom" && e.customType === GATE_REQUEST_ENTRY_TYPE; + return !!e && e.type === 'custom' && e.customType === GATE_REQUEST_ENTRY_TYPE; } export function isGateDecisionEntry(entry: unknown): boolean { const e = entry as CustomEntry | null; - return !!e && e.type === "custom" && e.customType === GATE_DECISION_ENTRY_TYPE; + return !!e && e.type === 'custom' && e.customType === GATE_DECISION_ENTRY_TYPE; } export function gateRequestFromEntry(entry: unknown): GateRequest | null { if (!isGateRequestEntry(entry)) return null; const d = (entry as CustomEntry).data as Record | undefined; - if (!d || typeof d.gateId !== "number" || typeof d.summary !== "string" || typeof d.proposed_action !== "string") { + if ( + !d || + typeof d.gateId !== 'number' || + typeof d.summary !== 'string' || + typeof d.proposed_action !== 'string' + ) { return null; } return { gateId: d.gateId, summary: d.summary, proposed_action: d.proposed_action }; @@ -72,8 +81,12 @@ export interface GateState { /** Derive gate state from the durable session entries (the .entry payloads from store.read). */ export function computeGateState(entries: unknown[]): GateState { - const gateRequests = entries.map(gateRequestFromEntry).filter((r): r is GateRequest => r !== null); - const gateDecisions = entries.map(gateDecisionFromEntry).filter((d): d is GateDecision => d !== null); + const gateRequests = entries + .map(gateRequestFromEntry) + .filter((r): r is GateRequest => r !== null); + const gateDecisions = entries + .map(gateDecisionFromEntry) + .filter((d): d is GateDecision => d !== null); const decidedIds = new Set(gateDecisions.map((d) => d.gateId)); // At most one gate is unanswered at a time (gates are sequential); pick the latest undecided. let pendingGate: GateRequest | null = null; @@ -81,33 +94,43 @@ export function computeGateState(entries: unknown[]): GateState { if (!decidedIds.has(r.gateId)) pendingGate = r; } const lastDecision = gateDecisions.length ? gateDecisions[gateDecisions.length - 1] : null; - return { gateRequests, gateDecisions, pendingGate, lastDecision, nextGateId: gateRequests.length }; + return { + gateRequests, + gateDecisions, + pendingGate, + lastDecision, + nextGateId: gateRequests.length, + }; } -export function continuationPrompt(action: "approve" | "reject", feedback?: string): string { - if (action === "approve") { +export function continuationPrompt(action: 'approve' | 'reject', feedback?: string): string { + if (action === 'approve') { return [ - `Human decision: APPROVED.${feedback ? ` ${feedback}` : ""}`, + `Human decision: APPROVED.${feedback ? ` ${feedback}` : ''}`, `Proceed with the proposed action. When finished, call submit_verdict exactly once, then stop.`, - ].join("\n"); + ].join('\n'); } return [ - `Human decision: REJECTED. ${feedback ?? "No feedback provided."}`, + `Human decision: REJECTED. ${feedback ?? 'No feedback provided.'}`, `Revise accordingly. You may call request_approval again when ready, or call submit_verdict when done.`, - ].join("\n"); + ].join('\n'); } export type SeedDecision = - | { kind: "paused"; gate: GateRequest } - | { kind: "abort"; record: GateDecision | null } - | { kind: "seed"; prompt: string; record: GateDecision | null }; + | { kind: 'paused'; gate: GateRequest } + | { kind: 'abort'; record: GateDecision | null } + | { kind: 'seed'; prompt: string; record: GateDecision | null }; /** * Decide what a runLeaf invocation should do, given the durable gate state, the (optional) decision * read from decisionRef, and the fresh job-mode prompt. Pure: the caller performs the side effects * (append `record`, run `session.prompt(prompt)`, set capture flags). See spec §3. */ -export function decideSeed(state: GateState, decision: Decision | null, freshPrompt: string): SeedDecision { +export function decideSeed( + state: GateState, + decision: Decision | null, + freshPrompt: string, +): SeedDecision { const { pendingGate, lastDecision, gateDecisions } = state; const decidedIds = new Set(gateDecisions.map((d) => d.gateId)); @@ -116,14 +139,20 @@ export function decideSeed(state: GateState, decision: Decision | null, freshPro const record: GateDecision | null = decidedIds.has(pendingGate.gateId) ? null : { gateId: decision.gateId, action: decision.action, feedback: decision.feedback }; - if (decision.action === "abort") return { kind: "abort", record }; - return { kind: "seed", prompt: continuationPrompt(decision.action, decision.feedback), record }; + if (decision.action === 'abort') return { kind: 'abort', record }; + return { + kind: 'seed', + prompt: continuationPrompt(decision.action, decision.feedback), + record, + }; } - return { kind: "paused", gate: pendingGate }; + return { kind: 'paused', gate: pendingGate }; } // No pending gate. A prior abort is terminal; otherwise re-derive the last continuation, else fresh. - if (lastDecision?.action === "abort") return { kind: "abort", record: null }; - const prompt = lastDecision ? continuationPrompt(lastDecision.action, lastDecision.feedback) : freshPrompt; - return { kind: "seed", prompt, record: null }; + if (lastDecision?.action === 'abort') return { kind: 'abort', record: null }; + const prompt = lastDecision + ? continuationPrompt(lastDecision.action, lastDecision.feedback) + : freshPrompt; + return { kind: 'seed', prompt, record: null }; } diff --git a/harness/src/index.ts b/harness/src/index.ts index 1d4342b..1acdbd9 100644 --- a/harness/src/index.ts +++ b/harness/src/index.ts @@ -1,2 +1,7 @@ -export { runTurn, type TurnConfig, type TurnResult } from "./run-turn.js"; -export { RedisRecordStore, recordsKey, type RecordStore, type SandboxRecord } from "./pool-records.js"; +export { runTurn, type TurnConfig, type TurnResult } from './run-turn.js'; +export { + RedisRecordStore, + recordsKey, + type RecordStore, + type SandboxRecord, +} from './pool-records.js'; diff --git a/harness/src/leaf-job-runner.ts b/harness/src/leaf-job-runner.ts index a0531b4..315edf2 100644 --- a/harness/src/leaf-job-runner.ts +++ b/harness/src/leaf-job-runner.ts @@ -1,8 +1,8 @@ // harness/src/leaf-job-runner.ts -import type { WorkQueue } from "@sh/work-queue"; -import { classifyOutcome } from "./classify-outcome.js"; -import { leafSessionId, type LeafEnvelope, type LeafResult } from "./run-leaf.js"; -import { toResultRecord, writeResult, type RedisLike } from "./leaf-result-store.js"; +import type { WorkQueue } from '@sh/work-queue'; +import { classifyOutcome } from './classify-outcome.js'; +import { leafSessionId, type LeafEnvelope, type LeafResult } from './run-leaf.js'; +import { toResultRecord, writeResult, type RedisLike } from './leaf-result-store.js'; export interface LeafJobDeps { queue: WorkQueue; @@ -30,29 +30,46 @@ export interface LeafJobDeps { */ export async function processOne( deps: LeafJobDeps, -): Promise<"done" | "failed" | "paused" | "aborted" | "solved" | "responded" | "deadletter" | "idle" | "retry"> { +): Promise< + | 'done' + | 'failed' + | 'paused' + | 'aborted' + | 'solved' + | 'responded' + | 'deadletter' + | 'idle' + | 'retry' +> { const maxAttempts = deps.maxAttempts ?? 3; const minIdleMs = deps.minIdleMs ?? 90_000; const blockMs = deps.blockMs ?? 5_000; const heartbeatMs = deps.heartbeatMs ?? 30_000; const now = deps.now ?? (() => new Date().toISOString()); const setHb = deps.setHeartbeat ?? ((fn, ms) => setInterval(fn, ms)); - const clearHb = deps.clearHeartbeat ?? ((h) => clearInterval(h as ReturnType)); + const clearHb = + deps.clearHeartbeat ?? ((h) => clearInterval(h as ReturnType)); const claimed = await deps.queue.claim(deps.consumerId, { minIdleMs, blockMs }); - if (!claimed) return "idle"; + if (!claimed) return 'idle'; const env = claimed.envelope as LeafEnvelope; const sid = leafSessionId(env); if (claimed.deliveryCount > maxAttempts) { - await writeResult(deps.resultStore, sid, - toResultRecord({ status: "failed", reason: "error" }, env.sessionId, now()), deps.ttlSeconds); + await writeResult( + deps.resultStore, + sid, + toResultRecord({ status: 'failed', reason: 'error' }, env.sessionId, now()), + deps.ttlSeconds, + ); await deps.queue.ack(claimed.entryId); - return "deadletter"; + return 'deadletter'; } - const hb = setHb(() => { void deps.queue.touch(claimed.entryId, deps.consumerId); }, heartbeatMs); + const hb = setHb(() => { + void deps.queue.touch(claimed.entryId, deps.consumerId); + }, heartbeatMs); let result: LeafResult; try { result = await deps.runLeaf(env); @@ -61,12 +78,17 @@ export async function processOne( } const outcome = classifyOutcome(result); - if (outcome.retryable) return "retry"; + if (outcome.retryable) return 'retry'; - await writeResult(deps.resultStore, sid, toResultRecord(result, env.sessionId, now()), deps.ttlSeconds); + await writeResult( + deps.resultStore, + sid, + toResultRecord(result, env.sessionId, now()), + deps.ttlSeconds, + ); if (outcome.ack) { await deps.queue.ack(claimed.entryId); return result.status; } - return "retry"; + return 'retry'; } diff --git a/harness/src/leaf-result-store.ts b/harness/src/leaf-result-store.ts index e2616e8..a73ba36 100644 --- a/harness/src/leaf-result-store.ts +++ b/harness/src/leaf-result-store.ts @@ -1,14 +1,14 @@ -import { createClient, type RedisClientType } from "redis"; -import type { Verdict } from "./verdict.js"; -import type { LeafResult, LeafUsage } from "./run-leaf.js"; +import { createClient, type RedisClientType } from 'redis'; +import type { Verdict } from './verdict.js'; +import type { LeafResult, LeafUsage } from './run-leaf.js'; export interface LeafResultRecord { - status: "done" | "failed" | "aborted" | "paused" | "solved" | "responded"; + status: 'done' | 'failed' | 'aborted' | 'paused' | 'solved' | 'responded'; verdict: Verdict | null; gate: { gateId: number; summary: string; proposed_action: string } | null; reason: string | null; - patch: string | null; // solve-leaf candidate patch (unified diff); null for non-solve results - text: string | null; // prompt-leaf assistant text; null for non-prompt results + patch: string | null; // solve-leaf candidate patch (unified diff); null for non-solve results + text: string | null; // prompt-leaf assistant text; null for non-prompt results usage: LeafUsage | null; // solve/prompt cumulative token usage (for run cost pricing); null otherwise sessionId: string; // RAW (un-sanitized) id, for caller correlation ts: string; @@ -25,33 +25,69 @@ export function resultKey(leafSessionId: string): string { } /** Map a terminal LeafResult to the persisted record. `rawSessionId` is the un-sanitized envelope id. */ -export function toResultRecord(result: LeafResult, rawSessionId: string, ts: string): LeafResultRecord { - const base: LeafResultRecord = { status: "failed", verdict: null, gate: null, reason: null, patch: null, text: null, usage: null, sessionId: rawSessionId, ts }; - if (result.status === "done") return { ...base, status: "done", verdict: result.verdict }; - if (result.status === "solved") return { ...base, status: "solved", patch: result.patch, usage: result.usage ?? null }; - if (result.status === "responded") return { ...base, status: "responded", text: result.text, usage: result.usage ?? null }; - if (result.status === "paused") { - return { ...base, status: "paused", gate: { gateId: result.gateId, summary: result.gate.summary, proposed_action: result.gate.proposed_action } }; +export function toResultRecord( + result: LeafResult, + rawSessionId: string, + ts: string, +): LeafResultRecord { + const base: LeafResultRecord = { + status: 'failed', + verdict: null, + gate: null, + reason: null, + patch: null, + text: null, + usage: null, + sessionId: rawSessionId, + ts, + }; + if (result.status === 'done') return { ...base, status: 'done', verdict: result.verdict }; + if (result.status === 'solved') + return { ...base, status: 'solved', patch: result.patch, usage: result.usage ?? null }; + if (result.status === 'responded') + return { ...base, status: 'responded', text: result.text, usage: result.usage ?? null }; + if (result.status === 'paused') { + return { + ...base, + status: 'paused', + gate: { + gateId: result.gateId, + summary: result.gate.summary, + proposed_action: result.gate.proposed_action, + }, + }; } - if (result.status === "aborted") return { ...base, status: "aborted" }; - return { ...base, status: "failed", reason: result.reason }; + if (result.status === 'aborted') return { ...base, status: 'aborted' }; + return { ...base, status: 'failed', reason: result.reason }; } -export async function writeResult(redis: RedisLike, leafSessionId: string, record: LeafResultRecord, ttlSeconds: number): Promise { +export async function writeResult( + redis: RedisLike, + leafSessionId: string, + record: LeafResultRecord, + ttlSeconds: number, +): Promise { await redis.set(resultKey(leafSessionId), JSON.stringify(record), { EX: ttlSeconds }); } -export async function readResult(redis: RedisLike, leafSessionId: string): Promise { +export async function readResult( + redis: RedisLike, + leafSessionId: string, +): Promise { const raw = await redis.get(resultKey(leafSessionId)); if (!raw) return null; - try { return JSON.parse(raw) as LeafResultRecord; } catch { return null; } + try { + return JSON.parse(raw) as LeafResultRecord; + } catch { + return null; + } } /** Real client used by the server and async worker. Reuses REDIS_URL, connects lazily. */ export class RedisResultStore implements RedisLike { private client: RedisClientType; private ready: Promise; - constructor(url = process.env.REDIS_URL ?? "redis://127.0.0.1:6379") { + constructor(url = process.env.REDIS_URL ?? 'redis://127.0.0.1:6379') { this.client = createClient({ url }) as RedisClientType; this.ready = this.client.connect().then(() => undefined); } diff --git a/harness/src/pool-records.ts b/harness/src/pool-records.ts index 412d7bd..3a12ea3 100644 --- a/harness/src/pool-records.ts +++ b/harness/src/pool-records.ts @@ -1,11 +1,11 @@ -import { createClient, type RedisClientType } from "redis"; +import { createClient, type RedisClientType } from 'redis'; export interface SandboxRecord { sandboxId: string; labels: Record; capabilities: string[]; capacityMax: number; - transport: "grpc"; + transport: 'grpc'; } export interface RecordStore { @@ -16,14 +16,14 @@ export interface RecordStore { /** Redis hash of grpc presence records: field = sandboxId, value = JSON(SandboxRecord). */ export function recordsKey(): string { - return "sh:sandbox:records"; + return 'sh:sandbox:records'; } /** node-redis-backed record store. Connects lazily; reuses REDIS_URL. */ export class RedisRecordStore implements RecordStore { private client: RedisClientType; private ready: Promise; - constructor(url = process.env.REDIS_URL ?? "redis://127.0.0.1:6379") { + constructor(url = process.env.REDIS_URL ?? 'redis://127.0.0.1:6379') { this.client = createClient({ url }) as RedisClientType; this.ready = this.client.connect().then(() => undefined); } diff --git a/harness/src/request-approval-tool.ts b/harness/src/request-approval-tool.ts index 84579ed..6b66f92 100644 --- a/harness/src/request-approval-tool.ts +++ b/harness/src/request-approval-tool.ts @@ -1,6 +1,6 @@ // harness/src/request-approval-tool.ts -import type { ExtensionAPI, ExtensionFactory } from "@earendil-works/pi-coding-agent"; -import { GATE_REQUEST_ENTRY_TYPE, type GateRequest } from "./gate.js"; +import type { ExtensionAPI, ExtensionFactory } from '@earendil-works/pi-coding-agent'; +import { GATE_REQUEST_ENTRY_TYPE, type GateRequest } from './gate.js'; export interface GateCapture { gate?: GateRequest; @@ -15,12 +15,18 @@ export interface GateSink { // Inline TypeBox-compatible schema (avoids importing typebox which is only in pi-fork's node_modules). // The `as any` cast on registerTool bypasses the TSchema constraint at compile time. const params = { - type: "object", + type: 'object', properties: { - summary: { type: "string", description: "A short summary of what you have done / decided so far (the human reads this)" }, - proposed_action: { type: "string", description: "The action you propose to take next, pending sign-off" }, + summary: { + type: 'string', + description: 'A short summary of what you have done / decided so far (the human reads this)', + }, + proposed_action: { + type: 'string', + description: 'The action you propose to take next, pending sign-off', + }, }, - required: ["summary", "proposed_action"], + required: ['summary', 'proposed_action'], }; /** @@ -36,24 +42,43 @@ export function requestApprovalExtension( ): ExtensionFactory { return (pi: ExtensionAPI) => { pi.registerTool({ - name: "request_approval", - label: "Request approval", + name: 'request_approval', + label: 'Request approval', description: "Request human approval before proceeding. Provide a summary of what you've done and the " + - "action you propose. Call this at most once, then stop; the session pauses and resumes with " + + 'action you propose. Call this at most once, then stop; the session pauses and resumes with ' + "the human's decision.", parameters: params, async execute(_id: string, args: unknown) { const summary = (args as { summary?: unknown })?.summary; const proposed_action = (args as { proposed_action?: unknown })?.proposed_action; - if (typeof summary !== "string" || summary.length === 0 || - typeof proposed_action !== "string" || proposed_action.length === 0) { - return { isError: true, content: [{ type: "text", text: "Invalid request_approval: summary and proposed_action must be non-empty strings" }] }; + if ( + typeof summary !== 'string' || + summary.length === 0 || + typeof proposed_action !== 'string' || + proposed_action.length === 0 + ) { + return { + isError: true, + content: [ + { + type: 'text', + text: 'Invalid request_approval: summary and proposed_action must be non-empty strings', + }, + ], + }; } const gate: GateRequest = { gateId: nextGateId, summary, proposed_action }; capture.gate = gate; sink?.appendCustomEntry(GATE_REQUEST_ENTRY_TYPE, gate); - return { content: [{ type: "text", text: "Approval requested; the session will pause and resume with the human decision." }] }; + return { + content: [ + { + type: 'text', + text: 'Approval requested; the session will pause and resume with the human decision.', + }, + ], + }; }, } as any); }; diff --git a/harness/src/run-leaf.ts b/harness/src/run-leaf.ts index e51c4dc..34d7699 100644 --- a/harness/src/run-leaf.ts +++ b/harness/src/run-leaf.ts @@ -5,22 +5,50 @@ import { SessionManager, SettingsManager, type FileEntry, -} from "@earendil-works/pi-coding-agent"; -import { RedisSessionBackend } from "@sh/session-backend"; -import { k8sSandboxExtension, KubectlTransport } from "@sh/k8s-sandbox"; -import { selectPoolSandbox, SandboxPoolSaturatedError, type SelectedSandbox } from "./select-sandbox.js"; -import { convergeWorkspace, cleanupWorkspace, captureWorkspaceDiff } from "./converge.js"; -import { setupSwebenchWorkspace, captureSwebenchDiff, cleanupSwebench, swebenchVenvDir, buildSwebenchSolvePrompt } from "./swebench-setup.js"; -import { executeTurn, resolveModelSelection, requireModel, applyModelGateway, sumBranchUsage, type TurnConfig, type TurnResult } from "./run-turn.js"; -import { BufferedRedisBackend } from "./buffered-redis-backend.js"; -import { flushExtension } from "./flush-extension.js"; -import { checkpointExtension } from "./checkpoint-extension.js"; -import { submitVerdictExtension, VERDICT_ENTRY_TYPE, type VerdictCapture } from "./submit-verdict-tool.js"; -import { verdictTerminationExtension } from "./verdict-termination-extension.js"; -import { validateVerdict, type Verdict } from "./verdict.js"; -import type { GateCapture } from "./request-approval-tool.js"; -import { computeGateState, decideSeed, validateDecision, type Decision, GATE_DECISION_ENTRY_TYPE } from "./gate.js"; -import { requestApprovalExtension } from "./request-approval-tool.js"; +} from '@earendil-works/pi-coding-agent'; +import { RedisSessionBackend } from '@sh/session-backend'; +import { k8sSandboxExtension, KubectlTransport } from '@sh/k8s-sandbox'; +import { + selectPoolSandbox, + SandboxPoolSaturatedError, + type SelectedSandbox, +} from './select-sandbox.js'; +import { convergeWorkspace, cleanupWorkspace, captureWorkspaceDiff } from './converge.js'; +import { + setupSwebenchWorkspace, + captureSwebenchDiff, + cleanupSwebench, + swebenchVenvDir, + buildSwebenchSolvePrompt, +} from './swebench-setup.js'; +import { + executeTurn, + resolveModelSelection, + requireModel, + applyModelGateway, + sumBranchUsage, + type TurnConfig, + type TurnResult, +} from './run-turn.js'; +import { BufferedRedisBackend } from './buffered-redis-backend.js'; +import { flushExtension } from './flush-extension.js'; +import { checkpointExtension } from './checkpoint-extension.js'; +import { + submitVerdictExtension, + VERDICT_ENTRY_TYPE, + type VerdictCapture, +} from './submit-verdict-tool.js'; +import { verdictTerminationExtension } from './verdict-termination-extension.js'; +import { validateVerdict, type Verdict } from './verdict.js'; +import type { GateCapture } from './request-approval-tool.js'; +import { + computeGateState, + decideSeed, + validateDecision, + type Decision, + GATE_DECISION_ENTRY_TYPE, +} from './gate.js'; +import { requestApprovalExtension } from './request-approval-tool.js'; /** * Recover a verdict from a persisted `verdict` custom session entry (written by @@ -30,7 +58,7 @@ import { requestApprovalExtension } from "./request-approval-tool.js"; */ export function verdictFromCustomEntry(entry: unknown): Verdict | null { const e = entry as { type?: string; customType?: string; data?: unknown } | null; - if (!e || e.type !== "custom" || e.customType !== VERDICT_ENTRY_TYPE) return null; + if (!e || e.type !== 'custom' || e.customType !== VERDICT_ENTRY_TYPE) return null; const r = validateVerdict(e.data); return r.ok ? r.value : null; } @@ -42,7 +70,7 @@ export function verdictFromCustomEntry(entry: unknown): Verdict | null { * so a retry/resume of the same envelope id maps to the same session. */ export function toSessionId(sessionId: string): string { - const cleaned = sessionId.replace(/[^A-Za-z0-9._-]/g, "-"); + const cleaned = sessionId.replace(/[^A-Za-z0-9._-]/g, '-'); // Trim leading/trailing non-alphanumerics with a linear scan instead of a `^…+|…+$` trim regex, // which CodeQL flags as polynomial (js/polynomial-redos) on inputs with many separators. const isAlnum = (c: number) => @@ -51,29 +79,34 @@ export function toSessionId(sessionId: string): string { let end = cleaned.length; while (start < end && !isAlnum(cleaned.charCodeAt(start))) start++; while (end > start && !isAlnum(cleaned.charCodeAt(end - 1))) end--; - return cleaned.slice(start, end) || "leaf"; + return cleaned.slice(start, end) || 'leaf'; } -export interface LeafItem { item_id: string; file: string; pattern: string; require_approval?: boolean } +export interface LeafItem { + item_id: string; + file: string; + pattern: string; + require_approval?: boolean; +} export interface LeafEnvelope { sessionId: string; /** Pool selected by the workload-facing control plane; falls back to the process default. */ sandboxPoolSelector?: string; - item: LeafItem; // inputs inline (was inputsRef) - decision?: Decision; // resume/approve only (was decisionRef) + item: LeafItem; // inputs inline (was inputsRef) + decision?: Decision; // resume/approve only (was decisionRef) model?: string; provider?: string; - workspaceRef?: string; // derived from the worktree in P2 when repoUrl+ref are given - repoUrl?: string; // P2: git remote to converge the sandbox repo copy from - ref?: string; // P2: commit/branch/tag the leaf's worktree is pinned to + workspaceRef?: string; // derived from the worktree in P2 when repoUrl+ref are given + repoUrl?: string; // P2: git remote to converge the sandbox repo copy from + ref?: string; // P2: commit/branch/tag the leaf's worktree is pinned to maxTurns?: number; - async?: boolean; // when true, the HTTP layer enqueues instead of running inline - tenant?: string; // namespaces the session id - kind?: "converge" | "solve" | "prompt"; // absent/"converge" => existing behavior; "solve" => runSolveLeaf - problemStatement?: string; // required when kind === "solve": the task the agent must implement - prompt?: string; // required when kind === "prompt": the free-form prompt to run - env_key?: string; // swebench solve: triggers the contained swebench-setup path (Plan C) + async?: boolean; // when true, the HTTP layer enqueues instead of running inline + tenant?: string; // namespaces the session id + kind?: 'converge' | 'solve' | 'prompt'; // absent/"converge" => existing behavior; "solve" => runSolveLeaf + problemStatement?: string; // required when kind === "solve": the task the agent must implement + prompt?: string; // required when kind === "prompt": the free-form prompt to run + env_key?: string; // swebench solve: triggers the contained swebench-setup path (Plan C) } /** Apply a request-scoped pool selector without mutating the process-wide environment. */ @@ -89,19 +122,29 @@ export function leafSessionId(env: { sessionId: string; tenant?: string }): stri /** True when a solve envelope carries a non-empty env_key, triggering the contained swebench path. */ export function isSwebenchEnvelope(env: { kind?: string; env_key?: string }): boolean { - return env.kind === "solve" && typeof env.env_key === "string" && env.env_key.length > 0; + return env.kind === 'solve' && typeof env.env_key === 'string' && env.env_key.length > 0; } // Cumulative token usage for a solve leaf (summed across all agent turns), for run cost pricing. -export type LeafUsage = { input: number; output: number; cacheRead: number; cacheWrite: number; total: number }; +export type LeafUsage = { + input: number; + output: number; + cacheRead: number; + cacheWrite: number; + total: number; +}; export type LeafResult = - | { status: "done"; verdict: Verdict } - | { status: "paused"; gateId: number; gate: { summary: string; proposed_action: string } } - | { status: "aborted" } - | { status: "solved"; patch: string; usage?: LeafUsage } - | { status: "responded"; text: string; usage?: LeafUsage } - | { status: "failed"; reason: "no_verdict" | "invalid_verdict" | "bad_inputs" | "error" | "saturated"; message?: string }; + | { status: 'done'; verdict: Verdict } + | { status: 'paused'; gateId: number; gate: { summary: string; proposed_action: string } } + | { status: 'aborted' } + | { status: 'solved'; patch: string; usage?: LeafUsage } + | { status: 'responded'; text: string; usage?: LeafUsage } + | { + status: 'failed'; + reason: 'no_verdict' | 'invalid_verdict' | 'bad_inputs' | 'error' | 'saturated'; + message?: string; + }; /** * Strip trailing "/" from a workspace root with a linear scan. @@ -146,7 +189,7 @@ export function buildLeafPrompt(item: LeafItem, workspaceRef?: string): string { `Report by calling the submit_verdict tool exactly once with item_id="${item.item_id}". Do not do anything else.`, ); } - return lines.join("\n"); + return lines.join('\n'); } export function buildSolvePrompt(problemStatement: string, workspaceRef: string): string { @@ -164,7 +207,7 @@ export function buildSolvePrompt(problemStatement: string, workspaceRef: string) ``, `Implement a fix by editing files under ${root}. When you are confident the fix is complete,`, `stop — do not ask questions and do not call any reporting tool.`, - ].join("\n"); + ].join('\n'); } export type LeafCapture = VerdictCapture & GateCapture; @@ -184,10 +227,19 @@ export type ProduceSolve = ( ) => Promise; export function validateItem(o: unknown): LeafItem | null { - if (typeof o !== "object" || o === null) return null; + if (typeof o !== 'object' || o === null) return null; const x = o as Record; - if (typeof x.item_id === "string" && typeof x.file === "string" && typeof x.pattern === "string") { - return { item_id: x.item_id, file: x.file, pattern: x.pattern, require_approval: x.require_approval === true }; + if ( + typeof x.item_id === 'string' && + typeof x.file === 'string' && + typeof x.pattern === 'string' + ) { + return { + item_id: x.item_id, + file: x.file, + pattern: x.pattern, + require_approval: x.require_approval === true, + }; } return null; } @@ -195,12 +247,16 @@ export function validateItem(o: unknown): LeafItem | null { export async function runLeaf( env: LeafEnvelope, config?: TurnConfig, - deps?: { produceVerdict?: ProduceVerdict; produceSolve?: ProduceSolve; executeTurn?: typeof executeTurn }, + deps?: { + produceVerdict?: ProduceVerdict; + produceSolve?: ProduceSolve; + executeTurn?: typeof executeTurn; + }, ): Promise { - if (env.kind === "solve") return runSolveLeaf(env, config, deps); - if (env.kind === "prompt") return runPromptLeaf(env, config, deps); + if (env.kind === 'solve') return runSolveLeaf(env, config, deps); + if (env.kind === 'prompt') return runPromptLeaf(env, config, deps); const item = validateItem(env.item); - if (!item) return { status: "failed", reason: "bad_inputs" }; + if (!item) return { status: 'failed', reason: 'bad_inputs' }; const capture: LeafCapture = {}; const produce = deps?.produceVerdict ?? realProduceVerdict; @@ -211,25 +267,29 @@ export async function runLeaf( // 503 Retry-After on it (spec §4.3), and classifyOutcome keeps it retryable for the async // path (drains as leases free). Every other throw is a generic "error". if (err instanceof SandboxPoolSaturatedError) { - return { status: "failed", reason: "saturated", message: err.message }; + return { status: 'failed', reason: 'saturated', message: err.message }; } - return { status: "failed", reason: "error", message: err instanceof Error ? err.message : String(err) }; + return { + status: 'failed', + reason: 'error', + message: err instanceof Error ? err.message : String(err), + }; } // Gate outcomes take precedence over verdict handling. - if (capture.aborted) return { status: "aborted" }; + if (capture.aborted) return { status: 'aborted' }; if (capture.gate) { return { - status: "paused", + status: 'paused', gateId: capture.gate.gateId, gate: { summary: capture.gate.summary, proposed_action: capture.gate.proposed_action }, }; } - if (!capture.verdict) return { status: "failed", reason: "no_verdict" }; + if (!capture.verdict) return { status: 'failed', reason: 'no_verdict' }; const v = validateVerdict(capture.verdict); - if (!v.ok) return { status: "failed", reason: "invalid_verdict", message: v.error }; - return { status: "done", verdict: v.value }; + if (!v.ok) return { status: 'failed', reason: 'invalid_verdict', message: v.error }; + return { status: 'done', verdict: v.value }; } export async function runSolveLeaf( @@ -237,16 +297,22 @@ export async function runSolveLeaf( config?: TurnConfig, deps?: { produceSolve?: ProduceSolve }, ): Promise { - if (!env.problemStatement || !env.repoUrl || !env.ref) return { status: "failed", reason: "bad_inputs" }; + if (!env.problemStatement || !env.repoUrl || !env.ref) + return { status: 'failed', reason: 'bad_inputs' }; const capture: SolveCapture = {}; const produce = deps?.produceSolve ?? realProduceSolve; try { await produce(env, config, capture); } catch (err) { - if (err instanceof SandboxPoolSaturatedError) return { status: "failed", reason: "saturated", message: err.message }; - return { status: "failed", reason: "error", message: err instanceof Error ? err.message : String(err) }; + if (err instanceof SandboxPoolSaturatedError) + return { status: 'failed', reason: 'saturated', message: err.message }; + return { + status: 'failed', + reason: 'error', + message: err instanceof Error ? err.message : String(err), + }; } - return { status: "solved", patch: capture.patch ?? "", usage: capture.usage }; + return { status: 'solved', patch: capture.patch ?? '', usage: capture.usage }; } async function runPromptLeaf( @@ -254,7 +320,7 @@ async function runPromptLeaf( config?: TurnConfig, deps?: { executeTurn?: typeof executeTurn }, ): Promise { - if (!env.prompt) return { status: "failed", reason: "bad_inputs" }; + if (!env.prompt) return { status: 'failed', reason: 'bad_inputs' }; const cwd = config?.cwd ?? process.cwd(); const sid = leafSessionId(env); const selection = resolveModelSelection({ @@ -275,25 +341,31 @@ async function runPromptLeaf( let selected: SelectedSandbox | null; try { selected = await selectPoolSandbox(sandboxEnvironment(env), cwd, sid, { - cap: Number(process.env.KAGENTI_SANDBOX_CAP ?? "20"), - ttlMs: Number(process.env.KAGENTI_SANDBOX_LEASE_TTL_MS ?? "60000"), - remoteSandbox: process.env.SH_REMOTE_SANDBOX === "1", + cap: Number(process.env.KAGENTI_SANDBOX_CAP ?? '20'), + ttlMs: Number(process.env.KAGENTI_SANDBOX_LEASE_TTL_MS ?? '60000'), + remoteSandbox: process.env.SH_REMOTE_SANDBOX === '1', }); } catch (err) { // Saturation stays a distinct transient signal, as for the other kinds: the sync /runs path // bounded-waits then 503s on it, and classifyOutcome keeps it retryable for the async queue. if (err instanceof SandboxPoolSaturatedError) { - return { status: "failed", reason: "saturated", message: err.message }; + return { status: 'failed', reason: 'saturated', message: err.message }; } - return { status: "failed", reason: "error", message: err instanceof Error ? err.message : String(err) }; + return { + status: 'failed', + reason: 'error', + message: err instanceof Error ? err.message : String(err), + }; } let heartbeat: ReturnType | undefined; try { if (selected) { - const hbMs = Number(process.env.KAGENTI_SANDBOX_HEARTBEAT_MS ?? "20000"); + const hbMs = Number(process.env.KAGENTI_SANDBOX_HEARTBEAT_MS ?? '20000'); const lease = selected; - heartbeat = setInterval(() => { void lease.heartbeat(); }, hbMs); + heartbeat = setInterval(() => { + void lease.heartbeat(); + }, hbMs); } const r: TurnResult = await exec({ prompt: env.prompt, @@ -303,13 +375,18 @@ async function runPromptLeaf( selection, sandbox: { config: selected?.config ?? null, transport: selected?.transport }, }); - if (r.stopReason === "aborted") return { status: "aborted" }; - if (r.stopReason === "error") return { status: "failed", reason: "error", message: r.errorMessage }; - return { status: "responded", text: r.response, usage: r.usage }; + if (r.stopReason === 'aborted') return { status: 'aborted' }; + if (r.stopReason === 'error') + return { status: 'failed', reason: 'error', message: r.errorMessage }; + return { status: 'responded', text: r.response, usage: r.usage }; } catch (err) { // A throw from the turn must not escape past the finally: a lease held past a crashed turn // shrinks pool capacity until its TTL expires. - return { status: "failed", reason: "error", message: err instanceof Error ? err.message : String(err) }; + return { + status: 'failed', + reason: 'error', + message: err instanceof Error ? err.message : String(err), + }; } finally { if (heartbeat) clearInterval(heartbeat); if (selected) await selected.release(); @@ -334,17 +411,18 @@ export const realProduceSolve: ProduceSolve = async (env, config, capture) => { // A solve leaf MUST have a real sandbox worktree — fail fast (before any Redis/session work) if the // pool is unconfigured. selectPoolSandbox returns null when no sandbox is configured (see select-sandbox.ts). const selected = await selectPoolSandbox(sandboxEnvironment(env), cwd, sid, { - cap: Number(process.env.KAGENTI_SANDBOX_CAP ?? "20"), - ttlMs: Number(process.env.KAGENTI_SANDBOX_LEASE_TTL_MS ?? "60000"), + cap: Number(process.env.KAGENTI_SANDBOX_CAP ?? '20'), + ttlMs: Number(process.env.KAGENTI_SANDBOX_LEASE_TTL_MS ?? '60000'), }); - if (!selected) throw new Error("solve leaf requires a configured sandbox pool"); + if (!selected) throw new Error('solve leaf requires a configured sandbox pool'); - const store = new RedisSessionBackend(config?.redisUrl ?? "redis://localhost:6379"); + const store = new RedisSessionBackend(config?.redisUrl ?? 'redis://localhost:6379'); const backend = new BufferedRedisBackend(store); const prior = await store.read(sid); - const sessionManager = prior.length > 0 - ? await SessionManager.openFromCheckpoint(sid, backend, cwd) - : SessionManager.create(cwd, undefined, { id: sid }, backend); + const sessionManager = + prior.length > 0 + ? await SessionManager.openFromCheckpoint(sid, backend, cwd) + : SessionManager.create(cwd, undefined, { id: sid }, backend); let heartbeat: ReturnType | undefined; try { @@ -354,11 +432,14 @@ export const realProduceSolve: ProduceSolve = async (env, config, capture) => { if (swebench) { const t0 = Date.now(); workspaceRef = await setupSwebenchWorkspace(transport, { - repoUrl: env.repoUrl!, baseCommit: env.ref!, envKey: env.env_key!, runId: sid, + repoUrl: env.repoUrl!, + baseCommit: env.ref!, + envKey: env.env_key!, + runId: sid, }); // Separate setup-duty from solve-duty (spec §4): the driver reads this line for setup ms, // and solve-duty = total exec-timing delta − setupMs. - const safeSid = sid.replace(/[\r\n]+/g, ""); + const safeSid = sid.replace(/[\r\n]+/g, ''); console.error(`[swebench-phase] sid=${safeSid} setupMs=${Date.now() - t0}`); } else { workspaceRef = await convergeWorkspace(transport, env.repoUrl!, env.ref!, sid); @@ -366,8 +447,10 @@ export const realProduceSolve: ProduceSolve = async (env, config, capture) => { // A solve leaf edits files in its worktree; point the agent's sandbox cwd at that worktree so the // model's edits (relative or absolute) land where captureWorkspaceDiff reads them. const agentConfig = { ...selected.config, podCwd: workspaceRef }; - const hbMs = Number(process.env.KAGENTI_SANDBOX_HEARTBEAT_MS ?? "20000"); - heartbeat = setInterval(() => { void selected.heartbeat(); }, hbMs); + const hbMs = Number(process.env.KAGENTI_SANDBOX_HEARTBEAT_MS ?? '20000'); + heartbeat = setInterval(() => { + void selected.heartbeat(); + }, hbMs); const agentDir = getAgentDir(); const settingsManager = SettingsManager.create(cwd, agentDir); @@ -392,7 +475,11 @@ export const realProduceSolve: ProduceSolve = async (env, config, capture) => { try { const prompt = swebench - ? buildSwebenchSolvePrompt(env.problemStatement!, workspaceRef, `${swebenchVenvDir(sid)}/bin/python`) + ? buildSwebenchSolvePrompt( + env.problemStatement!, + workspaceRef, + `${swebenchVenvDir(sid)}/bin/python`, + ) : buildSolvePrompt(env.problemStatement!, workspaceRef); await session.prompt(prompt); // Per-leaf token usage (cumulative across all solve turns) for run cost pricing. Sum assistant @@ -404,7 +491,9 @@ export const realProduceSolve: ProduceSolve = async (env, config, capture) => { } catch { /* usage is best-effort */ } - capture.patch = swebench ? await captureSwebenchDiff(transport, sid) : await captureWorkspaceDiff(transport, sid); + capture.patch = swebench + ? await captureSwebenchDiff(transport, sid) + : await captureWorkspaceDiff(transport, sid); } finally { await backend.flush(); } @@ -438,10 +527,10 @@ export const realProduceVerdict: ProduceVerdict = async (item, env, config, capt // Durable, resumable session keyed by the (sanitized) session id. BufferedRedisBackend drains // writes continuously, so a mid-run crash preserves progress up to the last drained entry. const sid = leafSessionId(env); - const store = new RedisSessionBackend(config?.redisUrl ?? "redis://localhost:6379"); + const store = new RedisSessionBackend(config?.redisUrl ?? 'redis://localhost:6379'); const backend = new BufferedRedisBackend(store); const isVerdictEntry = (e: unknown) => - (e as { type?: string }).type === "custom" && + (e as { type?: string }).type === 'custom' && (e as { customType?: string }).customType === VERDICT_ENTRY_TYPE; // Resume the session if one already exists under this id (retry / post-crash); otherwise create @@ -467,10 +556,10 @@ export const realProduceVerdict: ProduceVerdict = async (item, env, config, capt // --- P2: choose a sandbox pod (pool lease) before building the prompt/session. Placed after the // verdict fast-path so a recovered verdict does not lease a pod. Returns null ⇒ no sandbox // configured (local tools). Throws SandboxPoolSaturatedError when a configured pool is full. - const remoteSandbox = process.env.SH_REMOTE_SANDBOX === "1"; + const remoteSandbox = process.env.SH_REMOTE_SANDBOX === '1'; const selected = await selectPoolSandbox(sandboxEnvironment(env), cwd, sid, { - cap: Number(process.env.KAGENTI_SANDBOX_CAP ?? "20"), - ttlMs: Number(process.env.KAGENTI_SANDBOX_LEASE_TTL_MS ?? "60000"), + cap: Number(process.env.KAGENTI_SANDBOX_CAP ?? '20'), + ttlMs: Number(process.env.KAGENTI_SANDBOX_LEASE_TTL_MS ?? '60000'), remoteSandbox, }); const converging = selected != null && !!env.repoUrl && !!env.ref; @@ -495,8 +584,10 @@ export const realProduceVerdict: ProduceVerdict = async (item, env, config, capt } } if (selected) { - const hbMs = Number(process.env.KAGENTI_SANDBOX_HEARTBEAT_MS ?? "20000"); - heartbeat = setInterval(() => { void selected.heartbeat(); }, hbMs); + const hbMs = Number(process.env.KAGENTI_SANDBOX_HEARTBEAT_MS ?? '20000'); + heartbeat = setInterval(() => { + void selected.heartbeat(); + }, hbMs); } // Gate front-end (design §3): decide whether to pause, abort, or seed a prompt. @@ -505,13 +596,13 @@ export const realProduceVerdict: ProduceVerdict = async (item, env, config, capt const decision = dv && dv.ok ? dv.value : null; const seed = decideSeed(gateState, decision, buildLeafPrompt(item, workspaceRef)); - if (seed.kind === "abort") { + if (seed.kind === 'abort') { if (seed.record) sessionManager.appendCustomEntry(GATE_DECISION_ENTRY_TYPE, seed.record); capture.aborted = true; await backend.flush(); return; } - if (seed.kind === "paused") { + if (seed.kind === 'paused') { capture.gate = seed.gate; await backend.flush(); return; diff --git a/harness/src/run-turn.ts b/harness/src/run-turn.ts index c960312..76fc599 100644 --- a/harness/src/run-turn.ts +++ b/harness/src/run-turn.ts @@ -5,19 +5,30 @@ import { SessionManager, SettingsManager, type FileEntry, -} from "@earendil-works/pi-coding-agent"; -import { getModel, getModels, getProviders, type AssistantMessage, type Model } from "@earendil-works/pi-ai"; -import { RedisSessionBackend } from "@sh/session-backend"; -import { BufferedRedisBackend } from "./buffered-redis-backend.js"; -import { flushExtension } from "./flush-extension.js"; -import { k8sSandboxExtension, resolveSandboxConfig, type K8sSandboxConfig, type SandboxTransport } from "@sh/k8s-sandbox"; -import { checkpointExtension } from "./checkpoint-extension.js"; -import { budgetVoterExtension, branchSpend } from "./budget-voter.js"; -import { toolChoiceExtension } from "./tool-choice-extension.js"; +} from '@earendil-works/pi-coding-agent'; +import { + getModel, + getModels, + getProviders, + type AssistantMessage, + type Model, +} from '@earendil-works/pi-ai'; +import { RedisSessionBackend } from '@sh/session-backend'; +import { BufferedRedisBackend } from './buffered-redis-backend.js'; +import { flushExtension } from './flush-extension.js'; +import { + k8sSandboxExtension, + resolveSandboxConfig, + type K8sSandboxConfig, + type SandboxTransport, +} from '@sh/k8s-sandbox'; +import { checkpointExtension } from './checkpoint-extension.js'; +import { budgetVoterExtension, branchSpend } from './budget-voter.js'; +import { toolChoiceExtension } from './tool-choice-extension.js'; // Type-only import (erased at compile time) so it is safe against the run-leaf↔run-turn value // cycle: run-leaf.ts imports values from run-turn.js, but a `import type` adds no runtime edge. -import type { LeafUsage } from "./run-leaf.js"; -import { sseExtension, type TurnStreamFrame } from "./turn-stream.js"; +import type { LeafUsage } from './run-leaf.js'; +import { sseExtension, type TurnStreamFrame } from './turn-stream.js'; /** * The sandbox a turn's tool calls run in: a resolved pod/pool config (null ⇒ run tools in the @@ -65,8 +76,8 @@ export function resolveModelSelection( env: NodeJS.ProcessEnv = process.env, ): ModelSelection { return { - provider: config?.provider ?? env.SH_MODEL_PROVIDER ?? "anthropic", - modelId: config?.model ?? env.SH_MODEL ?? "claude-opus-4-8", + provider: config?.provider ?? env.SH_MODEL_PROVIDER ?? 'anthropic', + modelId: config?.model ?? env.SH_MODEL ?? 'claude-opus-4-8', }; } @@ -98,22 +109,27 @@ function parseModelHeaders(env: NodeJS.ProcessEnv): Record { try { parsed = JSON.parse(raw); } catch { - throw new Error(`SH_MODEL_HEADERS must be a JSON object of header name→value pairs (got: ${raw}).`); + throw new Error( + `SH_MODEL_HEADERS must be a JSON object of header name→value pairs (got: ${raw}).`, + ); } - if (!parsed || typeof parsed !== "object" || Array.isArray(parsed)) { + if (!parsed || typeof parsed !== 'object' || Array.isArray(parsed)) { throw new Error(`SH_MODEL_HEADERS must be a JSON object of header name→value pairs.`); } const out: Record = {}; for (const [k, v] of Object.entries(parsed as Record)) { out[k] = - typeof v === "string" - ? v.replace(/\$\{([A-Za-z_][A-Za-z0-9_]*)\}/g, (_m, name: string) => env[name] ?? "") + typeof v === 'string' + ? v.replace(/\$\{([A-Za-z_][A-Za-z0-9_]*)\}/g, (_m, name: string) => env[name] ?? '') : String(v); } return out; } -function synthesizeCustomModel(modelId: string, env: NodeJS.ProcessEnv): Model<"anthropic-messages"> { +function synthesizeCustomModel( + modelId: string, + env: NodeJS.ProcessEnv, +): Model<'anthropic-messages'> { // SH_MODEL_BASE_URL is the protocol-neutral knob; ANTHROPIC_BASE_URL is the back-compat fallback. const baseUrl = env.SH_MODEL_BASE_URL || env.ANTHROPIC_BASE_URL; if (!baseUrl) { @@ -126,20 +142,20 @@ function synthesizeCustomModel(modelId: string, env: NodeJS.ProcessEnv): Model<" // Typed as Model<"anthropic-messages"> (not `as ReturnType`) so tsc checks // the shape — if pi-ai's Model type gains a required field, this fails to compile instead of // silently omitting it. - const model: Model<"anthropic-messages"> = { + const model: Model<'anthropic-messages'> = { id: modelId, name: modelId, - api: "anthropic-messages", + api: 'anthropic-messages', // provider MUST be "anthropic" (not a synthetic tag): pi resolves the request API key by // provider name — authStorage.getApiKey(provider) maps "anthropic" -> ANTHROPIC_API_KEY // (which applyModelGateway seeds from the auth token), whereas an unknown provider like // "custom" has no env-key mapping and fails with `No API key found for "custom"`. Request // routing is by baseUrl + api, not provider, so tagging it "anthropic" sends traffic to the // custom baseUrl while satisfying the key lookup. Overridable via SH_MODEL_PROVIDER. - provider: (env.SH_MODEL_PROVIDER ?? "anthropic") as Model<"anthropic-messages">["provider"], + provider: (env.SH_MODEL_PROVIDER ?? 'anthropic') as Model<'anthropic-messages'>['provider'], baseUrl, reasoning: false, - input: ["text"], + input: ['text'], cost: { input: 0, output: 0, cacheRead: 0, cacheWrite: 0 }, contextWindow, maxTokens, @@ -163,7 +179,7 @@ function synthesizeCustomModel(modelId: string, env: NodeJS.ProcessEnv): Model<" function synthesizeOpenAICompletionsModel( modelId: string, env: NodeJS.ProcessEnv, -): Model<"openai-completions"> { +): Model<'openai-completions'> { const baseUrl = env.SH_MODEL_BASE_URL || env.OPENAI_BASE_URL; if (!baseUrl) { throw new Error( @@ -172,26 +188,26 @@ function synthesizeOpenAICompletionsModel( } const contextWindow = Number(env.SH_MODEL_CONTEXT_WINDOW) || 131072; const maxTokens = Number(env.SH_MODEL_MAX_TOKENS) || 8192; - const auth = env.SH_MODEL_AUTH ?? "bearer"; + const auth = env.SH_MODEL_AUTH ?? 'bearer'; const headers: Record = { ...parseModelHeaders(env) }; - if (auth === "custom-header" || auth === "none") { + if (auth === 'custom-header' || auth === 'none') { // Endpoint authenticates via a custom header (already in `headers`) or not at all — strip the // SDK's default Authorization Bearer so an unknown/empty Bearer isn't sent. pi's openai client // still requires a non-empty api key even when the Bearer is unused, so seed a placeholder. headers.Authorization = null; - if (!env.OPENAI_API_KEY) process.env.OPENAI_API_KEY = "unused"; + if (!env.OPENAI_API_KEY) process.env.OPENAI_API_KEY = 'unused'; } - const model: Model<"openai-completions"> = { + const model: Model<'openai-completions'> = { id: modelId, name: modelId, - api: "openai-completions", + api: 'openai-completions', // provider "openai" so pi resolves the api key from OPENAI_API_KEY (env-api-keys.ts). Request // routing is by baseUrl + api; provider only drives the key lookup. Overridable via SH_MODEL_PROVIDER. - provider: (env.SH_MODEL_PROVIDER ?? "openai") as Model<"openai-completions">["provider"], + provider: (env.SH_MODEL_PROVIDER ?? 'openai') as Model<'openai-completions'>['provider'], baseUrl, headers: headers as unknown as Record, reasoning: false, - input: ["text"], + input: ['text'], cost: { input: 0, output: 0, cacheRead: 0, cacheWrite: 0 }, contextWindow, maxTokens, @@ -219,15 +235,15 @@ export function requireModel( modelId: string, env: NodeJS.ProcessEnv = process.env, ) { - if (env.SH_MODEL_CUSTOM === "1") { - const api = env.SH_MODEL_API ?? "anthropic"; + if (env.SH_MODEL_CUSTOM === '1') { + const api = env.SH_MODEL_API ?? 'anthropic'; switch (api) { - case "anthropic": - case "anthropic-messages": + case 'anthropic': + case 'anthropic-messages': return synthesizeCustomModel(modelId, env); - case "openai-completions": + case 'openai-completions': return synthesizeOpenAICompletionsModel(modelId, env); - case "openai-responses": + case 'openai-responses': throw new Error( `SH_MODEL_API=openai-responses is not yet implemented (deferred; see docs/specs/2026-08-20-multi-protocol-model-provider-design.md §7). Use openai-completions.`, ); @@ -242,16 +258,16 @@ export function requireModel( const providers = getProviders() as string[]; if (!providers.includes(provider)) { throw new Error( - `Unknown model provider "${provider}". Known providers: ${providers.join(", ")}.`, + `Unknown model provider "${provider}". Known providers: ${providers.join(', ')}.`, ); } const ids = (getModels(provider as never) as Array<{ id: string }>).map((m) => m.id); // Surface the dot-vs-dash (or case) twin if one exists — the common mistake. - const norm = (s: string) => s.replace(/[.\-]/g, "").toLowerCase(); + const norm = (s: string) => s.replace(/[.\-]/g, '').toLowerCase(); const suggestions = ids.filter((id) => norm(id) === norm(modelId)); const hint = suggestions.length - ? `Did you mean: ${suggestions.join(", ")}?` - : `Known "${provider}" ids include: ${ids.slice(0, 12).join(", ")}${ids.length > 12 ? ", …" : ""}.`; + ? `Did you mean: ${suggestions.join(', ')}?` + : `Known "${provider}" ids include: ${ids.slice(0, 12).join(', ')}${ids.length > 12 ? ', …' : ''}.`; throw new Error(`Unknown model "${provider}/${modelId}" — not in the pi-ai registry. ${hint}`); } @@ -276,14 +292,14 @@ export interface TurnResult { */ export function applyModelGateway }>( baseModel: M, - config?: Pick, + config?: Pick, ): M { // The Anthropic gateway rewrite (Bearer + strip x-api-key + seed ANTHROPIC_API_KEY, and the // litellm compat-flag disables) applies ONLY to the Anthropic-messages path. OpenAI-compatible // models carry their own baseUrl/headers/auth from synthesizeOpenAICompletionsModel — leave // them untouched (else we'd clobber baseUrl with ANTHROPIC_BASE_URL and inject a wrong Bearer). const api = (baseModel as { api?: string }).api; - if (api && api !== "anthropic-messages") return baseModel; + if (api && api !== 'anthropic-messages') return baseModel; // `||` (not `??`) so an empty-string config value falls back to the env var rather than // suppressing it — "" is a "not set" sentinel here, not a meaningful credential. const authToken = config?.anthropicAuthToken || process.env.ANTHROPIC_AUTH_TOKEN; @@ -317,7 +333,7 @@ export function applyModelGateway headers: { ...baseModel.headers, Authorization: `Bearer ${authToken}`, - "x-api-key": null, // strip x-api-key when using gateway Bearer auth + 'x-api-key': null, // strip x-api-key when using gateway Bearer auth } as unknown as Record, } : {}), @@ -336,9 +352,12 @@ export function sumBranchUsage(sm: unknown): LeafUsage { const branch = (sm as { getBranch?: () => unknown[] }).getBranch?.() ?? []; for (const entry of branch as Array<{ type?: string; - message?: { role?: string; usage?: { input: number; output: number; cacheRead: number; cacheWrite: number } }; + message?: { + role?: string; + usage?: { input: number; output: number; cacheRead: number; cacheWrite: number }; + }; }>) { - if (entry?.type === "message" && entry.message?.role === "assistant" && entry.message.usage) { + if (entry?.type === 'message' && entry.message?.role === 'assistant' && entry.message.usage) { const m = entry.message.usage; u.input += m.input; u.output += m.output; @@ -370,7 +389,7 @@ export interface ExecuteTurnInput { */ export async function executeTurn(input: ExecuteTurnInput): Promise { const { prompt, sessionId, config, createIfAbsent } = input; - const redisUrl = config?.redisUrl ?? "redis://localhost:6379"; + const redisUrl = config?.redisUrl ?? 'redis://localhost:6379'; const cwd = config?.cwd ?? process.cwd(); const store = new RedisSessionBackend(redisUrl); @@ -385,7 +404,7 @@ export async function executeTurn(input: ExecuteTurnInput): Promise try { sessionManager = await SessionManager.openFromCheckpoint(sessionId, backend, cwd); } catch (err) { - if (err instanceof Error && err.message.includes("no session in backend")) { + if (err instanceof Error && err.message.includes('no session in backend')) { sessionManager = SessionManager.create(cwd, undefined, { id: sessionId }, backend); } else { throw err; @@ -410,10 +429,10 @@ export async function executeTurn(input: ExecuteTurnInput): Promise // Surface whether sandbox routing actually resolved: a null config means tool calls run in // the harness pod's own filesystem (local), not a sandbox pod — a common cause of "the file // never appeared in the sandbox". Cheap one-line signal in container logs. - if (process.env.SH_MODEL_CUSTOM === "1") { + if (process.env.SH_MODEL_CUSTOM === '1') { const how = sandbox.config - ? `${input.sandbox ? "leased" : "pod/pool"}${sandbox.transport ? " (grpc transport)" : ""}` - : "NULL (tools run LOCAL)"; + ? `${input.sandbox ? 'leased' : 'pod/pool'}${sandbox.transport ? ' (grpc transport)' : ''}` + : 'NULL (tools run LOCAL)'; console.error(`[sandbox] resolved config: ${how}`); } const extensionFactories = [ @@ -465,17 +484,17 @@ export async function executeTurn(input: ExecuteTurnInput): Promise await session.prompt(prompt); const lastMessage = session.state.messages.at(-1) as AssistantMessage | undefined; - let response = ""; - let stopReason = "end_turn"; + let response = ''; + let stopReason = 'end_turn'; let errorMessage: string | undefined; - if (lastMessage?.role === "assistant") { - stopReason = lastMessage.stopReason ?? "end_turn"; - if (stopReason === "error" || stopReason === "aborted") { + if (lastMessage?.role === 'assistant') { + stopReason = lastMessage.stopReason ?? 'end_turn'; + if (stopReason === 'error' || stopReason === 'aborted') { errorMessage = lastMessage.errorMessage || `Request ${stopReason}`; } else { for (const content of lastMessage.content) { - if (content.type === "text") { + if (content.type === 'text') { response += content.text; } } @@ -511,7 +530,7 @@ export function wireAbort(signal: AbortSignal, session: { abort: () => void }): session.abort(); return; } - signal.addEventListener("abort", () => session.abort(), { once: true }); + signal.addEventListener('abort', () => session.abort(), { once: true }); } /** diff --git a/harness/src/sandbox-lease.ts b/harness/src/sandbox-lease.ts index 30eb5b9..6565ac6 100644 --- a/harness/src/sandbox-lease.ts +++ b/harness/src/sandbox-lease.ts @@ -1,4 +1,4 @@ -import { createClient, type RedisClientType } from "redis"; +import { createClient, type RedisClientType } from 'redis'; /** Redis key holding the per-pod lease set (member = leaf id, score = expiry ms). */ export function leaseKey(pod: string): string { @@ -39,13 +39,16 @@ export interface LeaseStore { export class RedisLeaseStore implements LeaseStore { private client: RedisClientType; private ready: Promise; - constructor(url = process.env.REDIS_URL ?? "redis://127.0.0.1:6379", private now: () => number = Date.now) { + constructor( + url = process.env.REDIS_URL ?? 'redis://127.0.0.1:6379', + private now: () => number = Date.now, + ) { this.client = createClient({ url }) as RedisClientType; this.ready = this.client.connect().then(() => undefined); } async load(pod: string): Promise { await this.ready; - await this.client.zRemRangeByScore(leaseKey(pod), "-inf", this.now()); + await this.client.zRemRangeByScore(leaseKey(pod), '-inf', this.now()); return this.client.zCard(leaseKey(pod)); } async acquire(pod: string, cap: number, runId: string, ttlMs: number): Promise { diff --git a/harness/src/select-sandbox.ts b/harness/src/select-sandbox.ts index 71c0500..31a2a34 100644 --- a/harness/src/select-sandbox.ts +++ b/harness/src/select-sandbox.ts @@ -1,4 +1,4 @@ -import { credentials } from "@grpc/grpc-js"; +import { credentials } from '@grpc/grpc-js'; import { listPoolPods, resolveSandboxConfig, @@ -8,9 +8,9 @@ import { type K8sSandboxConfig, type SandboxTransport, type ExecClientLike, -} from "@sh/k8s-sandbox"; -import { RedisLeaseStore, type LeaseStore } from "./sandbox-lease.js"; -import { RedisRecordStore, type RecordStore, type SandboxRecord } from "./pool-records.js"; +} from '@sh/k8s-sandbox'; +import { RedisLeaseStore, type LeaseStore } from './sandbox-lease.js'; +import { RedisRecordStore, type RecordStore, type SandboxRecord } from './pool-records.js'; /** Pure: pods ordered ascending by active load (stable — ties keep input order). */ export function orderByLoad(loads: { pod: string; active: number }[]): string[] { @@ -24,7 +24,7 @@ export function orderByLoad(loads: { pod: string; active: number }[]): string[] export class SandboxPoolSaturatedError extends Error { constructor(selector: string) { super(`sandbox pool '${selector}' saturated: all pods at capacity`); - this.name = "SandboxPoolSaturatedError"; + this.name = 'SandboxPoolSaturatedError'; } } @@ -37,7 +37,12 @@ export interface SelectedSandbox { } export interface SelectDeps { - listPods?: (selector: string, namespace: string, context?: string, run?: RunKubectl) => Promise; + listPods?: ( + selector: string, + namespace: string, + context?: string, + run?: RunKubectl, + ) => Promise; lease?: LeaseStore; run?: RunKubectl; /** Mirrored grpc presence records; defaults to a RedisRecordStore. Only consulted when opts.remoteSandbox is true. */ @@ -48,7 +53,7 @@ export interface SelectDeps { /** Lazily builds a real gRPC exec client — only reached on the grpc branch when the flag is on. */ function defaultExecClient(_sandboxId: string, env: NodeJS.ProcessEnv): ExecClientLike { - const addr = env.SH_RELAY_ADDR ?? "sandbox-relay.default.svc.cluster.local:8443"; + const addr = env.SH_RELAY_ADDR ?? 'sandbox-relay.default.svc.cluster.local:8443'; return new SandboxExecClient(addr, credentials.createInsecure()) as unknown as ExecClientLike; } @@ -73,9 +78,9 @@ export async function selectPoolSandbox( return config ? { config, heartbeat: async () => {}, release: async () => {} } : null; } - const namespace = env.KAGENTI_SANDBOX_NAMESPACE ?? "default"; + const namespace = env.KAGENTI_SANDBOX_NAMESPACE ?? 'default'; const context = env.KAGENTI_SANDBOX_CONTEXT || undefined; - const podCwd = env.KAGENTI_SANDBOX_CWD ?? "/workspace"; + const podCwd = env.KAGENTI_SANDBOX_CWD ?? '/workspace'; const list = deps.listPods ?? listPoolPods; const lease = deps.lease ?? new RedisLeaseStore(env.REDIS_URL); @@ -103,13 +108,18 @@ export async function selectPoolSandbox( const candidates = [...pods, ...grpcRecs.map((r) => r.sandboxId)]; if (candidates.length === 0) throw new Error(`no Running pods for pool selector '${selector}'`); - const loads = await Promise.all(candidates.map(async (name) => ({ pod: name, active: await lease.load(name) }))); + const loads = await Promise.all( + candidates.map(async (name) => ({ pod: name, active: await lease.load(name) })), + ); for (const name of orderByLoad(loads)) { if (await lease.acquire(name, opts.cap, runId, opts.ttlMs)) { const config: K8sSandboxConfig = { pod: name, namespace, context, podCwd, headCwd }; const rec = grpcById.get(name); const transport = rec - ? GrpcRelayTransport(name, (deps.makeExecClient ?? ((id: string) => defaultExecClient(id, env)))(name)) + ? GrpcRelayTransport( + name, + (deps.makeExecClient ?? ((id: string) => defaultExecClient(id, env)))(name), + ) : undefined; return { config, diff --git a/harness/src/submit-verdict-tool.ts b/harness/src/submit-verdict-tool.ts index 5a80f89..ad29ab3 100644 --- a/harness/src/submit-verdict-tool.ts +++ b/harness/src/submit-verdict-tool.ts @@ -1,12 +1,12 @@ -import type { ExtensionAPI, ExtensionFactory } from "@earendil-works/pi-coding-agent"; -import { validateVerdict, type Verdict } from "./verdict.js"; +import type { ExtensionAPI, ExtensionFactory } from '@earendil-works/pi-coding-agent'; +import { validateVerdict, type Verdict } from './verdict.js'; export interface VerdictCapture { verdict?: Verdict; } /** Custom session-entry type used to persist a captured verdict durably (for resume recovery). */ -export const VERDICT_ENTRY_TYPE = "verdict"; +export const VERDICT_ENTRY_TYPE = 'verdict'; /** Minimal slice of SessionManager the tool needs to persist the verdict durably. */ export interface VerdictSink { @@ -16,16 +16,19 @@ export interface VerdictSink { // Inline TypeBox-compatible schema (avoids importing typebox which is only in pi-fork's node_modules). // The `as any` cast on registerTool bypasses the TSchema constraint at compile time. const params = { - type: "object", + type: 'object', properties: { - item_id: { type: "string", description: "The id of the item being judged (echo the input item_id)" }, + item_id: { + type: 'string', + description: 'The id of the item being judged (echo the input item_id)', + }, verdict: { - anyOf: [{ const: "FLAGGED" }, { const: "CLEAR" }], - description: "FLAGGED if the pattern is present and relevant; CLEAR otherwise", + anyOf: [{ const: 'FLAGGED' }, { const: 'CLEAR' }], + description: 'FLAGGED if the pattern is present and relevant; CLEAR otherwise', }, - reason: { type: "string", description: "One sentence justifying the verdict" }, + reason: { type: 'string', description: 'One sentence justifying the verdict' }, }, - required: ["item_id", "verdict", "reason"], + required: ['item_id', 'verdict', 'reason'], }; /** @@ -34,19 +37,25 @@ const params = { * session entry, so a session resumed after a crash can recover the verdict without re-running * the agent (mirrors the checkpoint-marker pattern). */ -export function submitVerdictExtension(capture: VerdictCapture, sink?: VerdictSink): ExtensionFactory { +export function submitVerdictExtension( + capture: VerdictCapture, + sink?: VerdictSink, +): ExtensionFactory { return (pi: ExtensionAPI) => { pi.registerTool({ - name: "submit_verdict", - label: "Submit verdict", + name: 'submit_verdict', + label: 'Submit verdict', description: - "Submit your final verdict for the item. Call this exactly once when you are done. " + - "After calling it, stop.", + 'Submit your final verdict for the item. Call this exactly once when you are done. ' + + 'After calling it, stop.', parameters: params, async execute(_id: string, args: unknown) { const r = validateVerdict(args); if (!r.ok) { - return { isError: true, content: [{ type: "text", text: `Invalid verdict: ${r.error}` }] }; + return { + isError: true, + content: [{ type: 'text', text: `Invalid verdict: ${r.error}` }], + }; } capture.verdict = r.value; sink?.appendCustomEntry(VERDICT_ENTRY_TYPE, r.value); @@ -54,7 +63,7 @@ export function submitVerdictExtension(capture: VerdictCapture, sink?: VerdictSi // (AgentToolResult.terminate) — without it the model just keeps getting re-prompted // and calls submit_verdict again, sometimes thousands of times, until something // external kills the pod. - return { content: [{ type: "text", text: "Verdict recorded." }], terminate: true }; + return { content: [{ type: 'text', text: 'Verdict recorded.' }], terminate: true }; }, } as any); }; diff --git a/harness/src/swebench-setup.ts b/harness/src/swebench-setup.ts index a24f3af..406d283 100644 --- a/harness/src/swebench-setup.ts +++ b/harness/src/swebench-setup.ts @@ -1,13 +1,19 @@ -import type { SandboxTransport } from "@sh/k8s-sandbox"; +import type { SandboxTransport } from '@sh/k8s-sandbox'; -function sq(s: string): string { return `'${s.replace(/'/g, `'\\''`)}'`; } +function sq(s: string): string { + return `'${s.replace(/'/g, `'\\''`)}'`; +} /** env_dir = env_key with a trailing ":latest" tag stripped (dots kept). */ export function envDirFromKey(envKey: string): string { - return envKey.endsWith(":latest") ? envKey.slice(0, -":latest".length) : envKey; + return envKey.endsWith(':latest') ? envKey.slice(0, -':latest'.length) : envKey; +} +export function swebenchCheckoutDir(runId: string): string { + return `/workspace/co-${runId}`; +} +export function swebenchVenvDir(runId: string): string { + return `/workspace/venv-${runId}`; } -export function swebenchCheckoutDir(runId: string): string { return `/workspace/co-${runId}`; } -export function swebenchVenvDir(runId: string): string { return `/workspace/venv-${runId}`; } /** * SWE-bench per-leaf provisioning inside the shared pool pod (mirrors the merged @@ -16,7 +22,12 @@ export function swebenchVenvDir(runId: string): string { return `/workspace/venv * /workspace EBS), checks out base_commit, layers a per-leaf system-site venv over the baked conda * env, and editable-installs the repo (build-iso fallback). Prints the checkout dir on stdout. */ -export function buildSwebenchSetupScript(a: { repoUrl: string; baseCommit: string; envKey: string; runId: string }): string { +export function buildSwebenchSetupScript(a: { + repoUrl: string; + baseCommit: string; + envKey: string; + runId: string; +}): string { const CO = swebenchCheckoutDir(a.runId); const VENV = swebenchVenvDir(a.runId); const ENV_PY = `/opt/miniconda3/envs/${envDirFromKey(a.envKey)}/bin/python`; @@ -31,19 +42,26 @@ export function buildSwebenchSetupScript(a: { repoUrl: string; baseCommit: strin `HOME=/workspace "$VENV/bin/pip" install -e "$CO" --no-build-isolation --no-cache-dir >&2 \\`, ` || HOME=/workspace "$VENV/bin/pip" install -e "$CO" --no-cache-dir >&2`, `printf '%s' "$CO"`, - ].join("\n"); + ].join('\n'); } export function buildSwebenchDiffScript(runId: string): string { const CO = swebenchCheckoutDir(runId); - return [`set -eu`, `git -C ${sq(CO)} add -A`, `git -C ${sq(CO)} diff --cached`].join("\n"); + return [`set -eu`, `git -C ${sq(CO)} add -A`, `git -C ${sq(CO)} diff --cached`].join('\n'); } export function buildSwebenchCleanupScript(runId: string): string { - return [`set -u`, `rm -rf ${sq(swebenchCheckoutDir(runId))} ${sq(swebenchVenvDir(runId))}`].join("\n"); + return [`set -u`, `rm -rf ${sq(swebenchCheckoutDir(runId))} ${sq(swebenchVenvDir(runId))}`].join( + '\n', + ); } -export function buildSwebenchSolvePrompt(problemStatement: string, checkoutDir: string, venvPython: string): string { - let end = checkoutDir.length; while (end > 0 && checkoutDir.charCodeAt(end - 1) === 47) end--; +export function buildSwebenchSolvePrompt( + problemStatement: string, + checkoutDir: string, + venvPython: string, +): string { + let end = checkoutDir.length; + while (end > 0 && checkoutDir.charCodeAt(end - 1) === 47) end--; const root = checkoutDir.slice(0, end); return [ `You are fixing a software issue in a checked-out Python repository.`, @@ -57,27 +75,40 @@ export function buildSwebenchSolvePrompt(problemStatement: string, checkoutDir: ``, `Implement a fix by editing files under ${root}. When you are confident the fix is complete,`, `stop — do not ask questions and do not call any reporting tool.`, - ].join("\n"); + ].join('\n'); } export async function setupSwebenchWorkspace( - t: SandboxTransport, a: { repoUrl: string; baseCommit: string; envKey: string; runId: string }, + t: SandboxTransport, + a: { repoUrl: string; baseCommit: string; envKey: string; runId: string }, ): Promise { - const { stdout, exitCode, truncated } = await t.exec(buildSwebenchSetupScript(a), { timeout: 900 }); + const { stdout, exitCode, truncated } = await t.exec(buildSwebenchSetupScript(a), { + timeout: 900, + }); if (truncated) { - throw new Error(`swebench setup exceeded the sandbox output cap (setup output too large): ${a.runId}`); + throw new Error( + `swebench setup exceeded the sandbox output cap (setup output too large): ${a.runId}`, + ); } if (exitCode !== 0) throw new Error(`swebench setup failed (exit ${exitCode})`); return stdout.toString().trim() || swebenchCheckoutDir(a.runId); } export async function captureSwebenchDiff(t: SandboxTransport, runId: string): Promise { - const { stdout, exitCode, truncated } = await t.exec(buildSwebenchDiffScript(runId), { timeout: 120 }); + const { stdout, exitCode, truncated } = await t.exec(buildSwebenchDiffScript(runId), { + timeout: 120, + }); if (truncated) { - throw new Error(`swebench diff capture exceeded the sandbox output cap (diff too large): ${runId}`); + throw new Error( + `swebench diff capture exceeded the sandbox output cap (diff too large): ${runId}`, + ); } if (exitCode !== 0) throw new Error(`swebench diff capture failed (exit ${exitCode})`); return stdout.toString(); } export async function cleanupSwebench(t: SandboxTransport, runId: string): Promise { - try { await t.exec(buildSwebenchCleanupScript(runId), { timeout: 60 }); } catch { /* ignore */ } + try { + await t.exec(buildSwebenchCleanupScript(runId), { timeout: 60 }); + } catch { + /* ignore */ + } } diff --git a/harness/src/tool-choice-extension.ts b/harness/src/tool-choice-extension.ts index cac931d..fcc7069 100644 --- a/harness/src/tool-choice-extension.ts +++ b/harness/src/tool-choice-extension.ts @@ -1,4 +1,4 @@ -import type { ExtensionFactory } from "@earendil-works/pi-coding-agent"; +import type { ExtensionFactory } from '@earendil-works/pi-coding-agent'; /** * For custom (non-Anthropic) model endpoints reached via SH_MODEL_CUSTOM=1, nudge tool use. @@ -17,20 +17,20 @@ import type { ExtensionFactory } from "@earendil-works/pi-coding-agent"; */ export function toolChoiceExtension(): ExtensionFactory { return (pi) => { - if (process.env.SH_MODEL_CUSTOM !== "1") return; + if (process.env.SH_MODEL_CUSTOM !== '1') return; let logged = false; // Handler receives the event { type, payload }; returning a value replaces the payload // (runner.ts before_provider_request contract). Mutate + return event.payload. - pi.on("before_provider_request", (event: { payload?: unknown }) => { + pi.on('before_provider_request', (event: { payload?: unknown }) => { const params = event.payload as Record | undefined; - if (!params || typeof params !== "object") return undefined; + if (!params || typeof params !== 'object') return undefined; const tools = params.tools as Array<{ name?: string }> | undefined; if (Array.isArray(tools) && tools.length > 0 && params.tool_choice == null) { // Emit the form the wire protocol accepts: Anthropic wants the object `{type:"auto"}`; // OpenAI Chat Completions (vLLM/RITS/OpenAI) wants the bare string "auto" and rejects the // object ("Invalid value for `function`: `None`"). Select on SH_MODEL_API (default anthropic). - const api = process.env.SH_MODEL_API ?? "anthropic"; - params.tool_choice = api.startsWith("openai") ? "auto" : { type: "auto" }; + const api = process.env.SH_MODEL_API ?? 'anthropic'; + params.tool_choice = api.startsWith('openai') ? 'auto' : { type: 'auto' }; } if (!logged) { logged = true; diff --git a/harness/src/turn-stream.ts b/harness/src/turn-stream.ts index c0d14a0..40250f0 100644 --- a/harness/src/turn-stream.ts +++ b/harness/src/turn-stream.ts @@ -1,6 +1,6 @@ -import type { ExtensionFactory } from "@earendil-works/pi-coding-agent"; -import type { LeafUsage } from "./run-leaf.js"; -import type { TurnResult } from "./run-turn.js"; +import type { ExtensionFactory } from '@earendil-works/pi-coding-agent'; +import type { LeafUsage } from './run-leaf.js'; +import type { TurnResult } from './run-turn.js'; /** * Neutral, transport-agnostic frames the turn core emits during a streamed turn. A discriminated @@ -8,12 +8,18 @@ import type { TurnResult } from "./run-turn.js"; * Fidelity B: tool_use carries verbatim args; tool_result carries isError + a clipped preview. */ export type TurnStreamFrame = - | { type: "text"; delta: string } // assistant-text token - | { type: "thinking"; delta: string } // reasoning token (optional; may never fire — §3.5) - | { type: "tool_use"; id: string; name: string; args: unknown } // tool call started (args verbatim) - | { type: "tool_result"; id: string; isError: boolean; preview: string } // tool call ended (clipped) - | { type: "done"; sessionId: string; stopReason: string; usage?: LeafUsage } - | { type: "error"; sessionId: string; stopReason: string; errorMessage?: string; usage?: LeafUsage }; + | { type: 'text'; delta: string } // assistant-text token + | { type: 'thinking'; delta: string } // reasoning token (optional; may never fire — §3.5) + | { type: 'tool_use'; id: string; name: string; args: unknown } // tool call started (args verbatim) + | { type: 'tool_result'; id: string; isError: boolean; preview: string } // tool call ended (clipped) + | { type: 'done'; sessionId: string; stopReason: string; usage?: LeafUsage } + | { + type: 'error'; + sessionId: string; + stopReason: string; + errorMessage?: string; + usage?: LeafUsage; + }; const DEFAULT_PREVIEW_BYTES = 2048; @@ -37,10 +43,10 @@ export function previewCap(override?: number): number { */ export function clip(result: unknown, previewBytes?: number): string { const cap = previewCap(previewBytes); - const text = typeof result === "string" ? result : (JSON.stringify(result) ?? ""); - const buf = Buffer.from(text, "utf8"); + const text = typeof result === 'string' ? result : (JSON.stringify(result) ?? ''); + const buf = Buffer.from(text, 'utf8'); if (buf.byteLength <= cap) return text; - return buf.subarray(0, cap).toString("utf8") + "…[truncated]"; + return buf.subarray(0, cap).toString('utf8') + '…[truncated]'; } /** @@ -53,17 +59,18 @@ export function sseExtension( opts?: { previewBytes?: number }, ): ExtensionFactory { return (pi) => { - pi.on("message_update", (e) => { + pi.on('message_update', (e) => { const a = e.assistantMessageEvent; - if (a.type === "text_delta" && a.delta) onEvent({ type: "text", delta: a.delta }); - else if (a.type === "thinking_delta" && a.delta) onEvent({ type: "thinking", delta: a.delta }); + if (a.type === 'text_delta' && a.delta) onEvent({ type: 'text', delta: a.delta }); + else if (a.type === 'thinking_delta' && a.delta) + onEvent({ type: 'thinking', delta: a.delta }); }); - pi.on("tool_execution_start", (e) => - onEvent({ type: "tool_use", id: e.toolCallId, name: e.toolName, args: e.args }), + pi.on('tool_execution_start', (e) => + onEvent({ type: 'tool_use', id: e.toolCallId, name: e.toolName, args: e.args }), ); - pi.on("tool_execution_end", (e) => + pi.on('tool_execution_end', (e) => onEvent({ - type: "tool_result", + type: 'tool_result', id: e.toolCallId, isError: e.isError, preview: clip(e.result, opts?.previewBytes), @@ -78,17 +85,17 @@ export function sseExtension( * streamed client ends with the same facts a sync client reads; only the event NAME differs. */ export function terminalFrame(result: TurnResult): TurnStreamFrame { - const clean = result.stopReason === "end_turn" || result.stopReason === "max_tokens"; + const clean = result.stopReason === 'end_turn' || result.stopReason === 'max_tokens'; if (clean) { return { - type: "done", + type: 'done', sessionId: result.sessionId, stopReason: result.stopReason, ...(result.usage ? { usage: result.usage } : {}), }; } return { - type: "error", + type: 'error', sessionId: result.sessionId, stopReason: result.stopReason, ...(result.errorMessage ? { errorMessage: result.errorMessage } : {}), diff --git a/harness/src/verdict-termination-extension.ts b/harness/src/verdict-termination-extension.ts index 773eb97..5b80dd3 100644 --- a/harness/src/verdict-termination-extension.ts +++ b/harness/src/verdict-termination-extension.ts @@ -1,5 +1,5 @@ -import type { ExtensionAPI, ExtensionFactory } from "@earendil-works/pi-coding-agent"; -import type { VerdictCapture } from "./submit-verdict-tool.js"; +import type { ExtensionAPI, ExtensionFactory } from '@earendil-works/pi-coding-agent'; +import type { VerdictCapture } from './submit-verdict-tool.js'; export interface TurnCounter { turns: number; @@ -37,16 +37,17 @@ export function verdictTerminationExtension( // Treat a missing OR non-positive maxTurns as "use the default" — a nullish-coalesce alone would // let an explicit 0 (or negative) through, which then fails the `> 0` guard below and silently // DISABLES the cap (unbounded), the opposite of what a caller passing 0 would expect. - const maxTurns = typeof opts.maxTurns === "number" && opts.maxTurns > 0 ? opts.maxTurns : DEFAULT_MAX_TURNS; + const maxTurns = + typeof opts.maxTurns === 'number' && opts.maxTurns > 0 ? opts.maxTurns : DEFAULT_MAX_TURNS; return (pi: ExtensionAPI) => { - pi.on("turn_start", () => { + pi.on('turn_start', () => { counter.turns += 1; }); - pi.on("tool_call", (event) => { + pi.on('tool_call', (event) => { if (capture.verdict) { - return { block: true, reason: "Verdict already submitted for this item — task complete." }; + return { block: true, reason: 'Verdict already submitted for this item — task complete.' }; } - if (typeof maxTurns === "number" && maxTurns > 0 && counter.turns > maxTurns) { + if (typeof maxTurns === 'number' && maxTurns > 0 && counter.turns > maxTurns) { return { block: true, reason: `Turn limit (${maxTurns}) reached without a submitted verdict — stopping.`, diff --git a/harness/src/verdict.ts b/harness/src/verdict.ts index d2394ac..d275a8e 100644 --- a/harness/src/verdict.ts +++ b/harness/src/verdict.ts @@ -1,4 +1,4 @@ -export type VerdictLabel = "FLAGGED" | "CLEAR"; +export type VerdictLabel = 'FLAGGED' | 'CLEAR'; export interface Verdict { item_id: string; @@ -9,18 +9,18 @@ export interface Verdict { export function validateVerdict( obj: unknown, ): { ok: true; value: Verdict } | { ok: false; error: string } { - if (typeof obj !== "object" || obj === null) { - return { ok: false, error: "verdict must be an object" }; + if (typeof obj !== 'object' || obj === null) { + return { ok: false, error: 'verdict must be an object' }; } const o = obj as Record; - if (typeof o.item_id !== "string" || o.item_id.length === 0) { - return { ok: false, error: "item_id must be a non-empty string" }; + if (typeof o.item_id !== 'string' || o.item_id.length === 0) { + return { ok: false, error: 'item_id must be a non-empty string' }; } - if (o.verdict !== "FLAGGED" && o.verdict !== "CLEAR") { + if (o.verdict !== 'FLAGGED' && o.verdict !== 'CLEAR') { return { ok: false, error: 'verdict must be "FLAGGED" or "CLEAR"' }; } - if (typeof o.reason !== "string") { - return { ok: false, error: "reason must be a string" }; + if (typeof o.reason !== 'string') { + return { ok: false, error: 'reason must be a string' }; } return { ok: true, value: { item_id: o.item_id, verdict: o.verdict, reason: o.reason } }; } diff --git a/harness/test/budget-voter.test.ts b/harness/test/budget-voter.test.ts index 4e8e60b..943996f 100644 --- a/harness/test/budget-voter.test.ts +++ b/harness/test/budget-voter.test.ts @@ -1,24 +1,24 @@ -import { describe, it, expect } from "vitest"; -import { - decideBudget, - sessionSpendTotal, - budgetVoterExtension, -} from "../src/budget-voter"; +import { describe, it, expect } from 'vitest'; +import { decideBudget, sessionSpendTotal, budgetVoterExtension } from '../src/budget-voter'; // --- minimal fakes (no Pi runtime) ------------------------------------------- -function fakeCtx(assistantUsages: Array<{ input: number; output: number; cacheRead: number; cacheWrite: number }>) { +function fakeCtx( + assistantUsages: Array<{ input: number; output: number; cacheRead: number; cacheWrite: number }>, +) { return { sessionManager: { getBranch: () => assistantUsages.map((usage) => ({ - type: "message", - message: { role: "assistant", usage }, + type: 'message', + message: { role: 'assistant', usage }, })), }, } as unknown as Parameters[0]; } function emptyCtx() { - return { sessionManager: { getBranch: () => [] } } as unknown as Parameters[0]; + return { sessionManager: { getBranch: () => [] } } as unknown as Parameters< + typeof sessionSpendTotal + >[0]; } function brokenCtx() { return {} as unknown as Parameters[0]; @@ -27,42 +27,51 @@ function brokenCtx() { function harness() { const handlers: Record = {}; const appended: Array<{ customType: string; data: unknown }> = []; - const pi = { on: (ev: string, h: Function) => { handlers[ev] = h; } }; - const sm = { appendCustomEntry: (customType: string, data: unknown) => { appended.push({ customType, data }); return "id"; } }; + const pi = { + on: (ev: string, h: Function) => { + handlers[ev] = h; + }, + }; + const sm = { + appendCustomEntry: (customType: string, data: unknown) => { + appended.push({ customType, data }); + return 'id'; + }, + }; return { handlers, appended, pi, sm }; } -describe("decideBudget", () => { - it("commits below the cap", () => { - expect(decideBudget({ spent: 10, estimated: 0, limit: 100 })).toEqual({ decision: "commit" }); +describe('decideBudget', () => { + it('commits below the cap', () => { + expect(decideBudget({ spent: 10, estimated: 0, limit: 100 })).toEqual({ decision: 'commit' }); }); - it("aborts when spent + estimated exceeds the cap", () => { + it('aborts when spent + estimated exceeds the cap', () => { expect(decideBudget({ spent: 90, estimated: 20, limit: 100 })).toEqual({ - decision: "abort", - reason: "budget_exceeded", + decision: 'abort', + reason: 'budget_exceeded', }); }); - it("commits exactly at the cap (not strictly greater)", () => { - expect(decideBudget({ spent: 100, estimated: 0, limit: 100 })).toEqual({ decision: "commit" }); + it('commits exactly at the cap (not strictly greater)', () => { + expect(decideBudget({ spent: 100, estimated: 0, limit: 100 })).toEqual({ decision: 'commit' }); }); - it("is disabled (always commits) when limit <= 0 or non-finite", () => { - expect(decideBudget({ spent: 999, estimated: 0, limit: 0 })).toEqual({ decision: "commit" }); - expect(decideBudget({ spent: 999, estimated: 0, limit: NaN })).toEqual({ decision: "commit" }); + it('is disabled (always commits) when limit <= 0 or non-finite', () => { + expect(decideBudget({ spent: 999, estimated: 0, limit: 0 })).toEqual({ decision: 'commit' }); + expect(decideBudget({ spent: 999, estimated: 0, limit: NaN })).toEqual({ decision: 'commit' }); }); }); -describe("sessionSpendTotal", () => { - it("sums all assistant usage fields", () => { +describe('sessionSpendTotal', () => { + it('sums all assistant usage fields', () => { const ctx = fakeCtx([ { input: 10, output: 5, cacheRead: 1, cacheWrite: 0 }, { input: 20, output: 4, cacheRead: 0, cacheWrite: 1 }, ]); expect(sessionSpendTotal(ctx)).toBe(41); }); - it("returns 0 for an empty branch (valid baseline, not null)", () => { + it('returns 0 for an empty branch (valid baseline, not null)', () => { expect(sessionSpendTotal(emptyCtx())).toBe(0); }); - it("returns null when the branch is unavailable", () => { + it('returns null when the branch is unavailable', () => { expect(sessionSpendTotal(brokenCtx())).toBeNull(); }); }); @@ -70,47 +79,54 @@ describe("sessionSpendTotal", () => { // The voter must work WITHOUT a session_start event: the headless runTurn path never // emits session_start (only bindExtensions/interactive/print/rpc do), so the baseline is // supplied by the caller via opts.baseline. These tests never fire session_start. -describe("budgetVoterExtension (headless: no session_start, injected baseline)", () => { - it("blocks a tool call and appends exactly one abort entry once over cap", () => { +describe('budgetVoterExtension (headless: no session_start, injected baseline)', () => { + it('blocks a tool call and appends exactly one abort entry once over cap', () => { const { handlers, appended, pi, sm } = harness(); budgetVoterExtension(sm as never, { limit: 50, baseline: 0 })(pi as never); // tool_call after 60 tokens of spend -> over the cap of 50 (no session_start fired) const ctx = fakeCtx([{ input: 40, output: 20, cacheRead: 0, cacheWrite: 0 }]); const res = handlers.tool_call({}, ctx); - expect(res).toEqual({ block: true, reason: "Session token budget exceeded" }); + expect(res).toEqual({ block: true, reason: 'Session token budget exceeded' }); expect(appended).toEqual([ - { customType: "abort", data: { reason: "budget_exceeded", spent: 60, limit: 50 } }, + { customType: 'abort', data: { reason: 'budget_exceeded', spent: 60, limit: 50 } }, ]); }); - it("does not block below the cap and writes no abort entry", () => { + it('does not block below the cap and writes no abort entry', () => { const { handlers, appended, pi, sm } = harness(); budgetVoterExtension(sm as never, { limit: 1000, baseline: 0 })(pi as never); const ctx = fakeCtx([{ input: 10, output: 5, cacheRead: 0, cacheWrite: 0 }]); expect(handlers.tool_call({}, ctx)).toEqual({}); expect(appended).toEqual([]); }); - it("subtracts the injected baseline so pre-turn spend is excluded", () => { + it('subtracts the injected baseline so pre-turn spend is excluded', () => { const { handlers, appended, pi, sm } = harness(); budgetVoterExtension(sm as never, { limit: 50, baseline: 100 })(pi as never); // total 130 -> this-turn spend 30 (< 50): no block - expect(handlers.tool_call({}, fakeCtx([{ input: 130, output: 0, cacheRead: 0, cacheWrite: 0 }]))).toEqual({}); + expect( + handlers.tool_call({}, fakeCtx([{ input: 130, output: 0, cacheRead: 0, cacheWrite: 0 }])), + ).toEqual({}); expect(appended).toEqual([]); // total 160 -> this-turn spend 60 (> 50): block once - const res = handlers.tool_call({}, fakeCtx([{ input: 160, output: 0, cacheRead: 0, cacheWrite: 0 }])); - expect(res).toEqual({ block: true, reason: "Session token budget exceeded" }); + const res = handlers.tool_call( + {}, + fakeCtx([{ input: 160, output: 0, cacheRead: 0, cacheWrite: 0 }]), + ); + expect(res).toEqual({ block: true, reason: 'Session token budget exceeded' }); expect(appended).toEqual([ - { customType: "abort", data: { reason: "budget_exceeded", spent: 60, limit: 50 } }, + { customType: 'abort', data: { reason: 'budget_exceeded', spent: 60, limit: 50 } }, ]); }); - it("does not block when the stat reading is unavailable (defensive)", () => { + it('does not block when the stat reading is unavailable (defensive)', () => { const { handlers, pi, sm } = harness(); budgetVoterExtension(sm as never, { limit: 1, baseline: 0 })(pi as never); expect(handlers.tool_call({}, brokenCtx())).toEqual({}); // total == null -> no block }); - it("is disabled when limit <= 0", () => { + it('is disabled when limit <= 0', () => { const { handlers, appended, pi, sm } = harness(); budgetVoterExtension(sm as never, { limit: 0, baseline: 0 })(pi as never); - expect(handlers.tool_call({}, fakeCtx([{ input: 10_000, output: 0, cacheRead: 0, cacheWrite: 0 }]))).toEqual({}); + expect( + handlers.tool_call({}, fakeCtx([{ input: 10_000, output: 0, cacheRead: 0, cacheWrite: 0 }])), + ).toEqual({}); expect(appended).toEqual([]); }); }); diff --git a/harness/test/buffered-redis-backend.test.ts b/harness/test/buffered-redis-backend.test.ts index f5db707..da56b59 100644 --- a/harness/test/buffered-redis-backend.test.ts +++ b/harness/test/buffered-redis-backend.test.ts @@ -1,13 +1,13 @@ -import { describe, it, expect } from "vitest"; -import { makeStoredEntry, type LogStore, type StoredEntry } from "@sh/session-backend"; -import type { FileEntry } from "@earendil-works/pi-coding-agent"; -import { BufferedRedisBackend } from "../src/buffered-redis-backend"; +import { describe, it, expect } from 'vitest'; +import { makeStoredEntry, type LogStore, type StoredEntry } from '@sh/session-backend'; +import type { FileEntry } from '@earendil-works/pi-coding-agent'; +import { BufferedRedisBackend } from '../src/buffered-redis-backend'; // The decorator only reads `.type` / `.customType` at runtime, so minimal cast objects // stand in for full FileEntry values. Helpers keep the casts in one place. -const msg = (): FileEntry => ({ type: "message" }) as unknown as FileEntry; +const msg = (): FileEntry => ({ type: 'message' }) as unknown as FileEntry; const custom = (customType: string): FileEntry => - ({ type: "custom", customType }) as unknown as FileEntry; + ({ type: 'custom', customType }) as unknown as FileEntry; const ct = (e: FileEntry): string | undefined => (e as { customType?: string }).customType; /** In-memory LogStore with a per-append delay to test ordering/async. */ @@ -15,51 +15,67 @@ class FakeStore implements LogStore { rows: StoredEntry[] = []; failNext = false; async append(sid: string, entry: FileEntry, piType: string): Promise> { - if (this.failNext) { this.failNext = false; throw new Error("boom"); } + if (this.failNext) { + this.failNext = false; + throw new Error('boom'); + } await new Promise((r) => setTimeout(r, 1)); - const stored = makeStoredEntry({ position: this.rows.length + 1, session_id: sid, piType, entry }); + const stored = makeStoredEntry({ + position: this.rows.length + 1, + session_id: sid, + piType, + entry, + }); this.rows.push(stored); return stored; } async read(_sid: string, from = 1): Promise[]> { return this.rows.filter((r) => r.position >= from); } - async latestWhere(_sid: string, p: (e: FileEntry) => boolean): Promise | null> { - const m = this.rows.filter((r) => p(r.entry)); return m.length ? m[m.length - 1] : null; + async latestWhere( + _sid: string, + p: (e: FileEntry) => boolean, + ): Promise | null> { + const m = this.rows.filter((r) => p(r.entry)); + return m.length ? m[m.length - 1] : null; + } + async nextPosition(): Promise { + return this.rows.length + 1; + } + async list(): Promise { + return ['s']; } - async nextPosition(): Promise { return this.rows.length + 1; } - async list(): Promise { return ["s"]; } } -describe("BufferedRedisBackend", () => { - it("append is fire-and-forget (resolves before the write completes) and preserves order on flush", async () => { +describe('BufferedRedisBackend', () => { + it('append is fire-and-forget (resolves before the write completes) and preserves order on flush', async () => { const store = new FakeStore(); const b = new BufferedRedisBackend(store); - b.append("s", custom("a")); - b.append("s", custom("b")); - b.append("s", custom("c")); + b.append('s', custom('a')); + b.append('s', custom('b')); + b.append('s', custom('c')); expect(store.rows.length).toBe(0); // nothing drained yet — writes are async await b.flush(); - expect(store.rows.map((r) => ct(r.entry))).toEqual(["a", "b", "c"]); + expect(store.rows.map((r) => ct(r.entry))).toEqual(['a', 'b', 'c']); }); - it("flush surfaces a write error from the queue", async () => { + it('flush surfaces a write error from the queue', async () => { const store = new FakeStore(); store.failNext = true; const b = new BufferedRedisBackend(store); - b.append("s", msg()); - await expect(b.flush()).rejects.toThrow("boom"); + b.append('s', msg()); + await expect(b.flush()).rejects.toThrow('boom'); }); - it("read unwraps stored entries; latestCheckpoint finds the checkpoint custom entry", async () => { + it('read unwraps stored entries; latestCheckpoint finds the checkpoint custom entry', async () => { const store = new FakeStore(); const b = new BufferedRedisBackend(store); - b.append("s", msg()); - b.append("s", custom("checkpoint")); + b.append('s', msg()); + b.append('s', custom('checkpoint')); await b.flush(); - const entries = await b.read("s"); - expect(entries.map((e) => e.type)).toEqual(["message", "custom"]); - const cp = await b.latestCheckpoint("s"); - expect(ct(cp as FileEntry)).toBe("checkpoint"); + const entries = await b.read('s'); + expect(entries.map((e) => e.type)).toEqual(['message', 'custom']); + const cp = await b.latestCheckpoint('s'); + expect(ct(cp as FileEntry)).toBe('checkpoint'); }); }); diff --git a/harness/test/checkpoint.test.ts b/harness/test/checkpoint.test.ts index 5ba07b8..e3d5e6f 100644 --- a/harness/test/checkpoint.test.ts +++ b/harness/test/checkpoint.test.ts @@ -1,10 +1,10 @@ -import { describe, it, expect, afterAll } from "vitest"; -import { SessionManager, type FileEntry } from "@earendil-works/pi-coding-agent"; -import { RedisSessionBackend } from "@sh/session-backend"; -import { BufferedRedisBackend } from "../src/buffered-redis-backend"; -import { checkpointExtension } from "../src/checkpoint-extension"; +import { describe, it, expect, afterAll } from 'vitest'; +import { SessionManager, type FileEntry } from '@earendil-works/pi-coding-agent'; +import { RedisSessionBackend } from '@sh/session-backend'; +import { BufferedRedisBackend } from '../src/buffered-redis-backend'; +import { checkpointExtension } from '../src/checkpoint-extension'; -const REDIS = process.env.REDIS_URL ?? "redis://127.0.0.1:6379"; +const REDIS = process.env.REDIS_URL ?? 'redis://127.0.0.1:6379'; const store = new RedisSessionBackend(REDIS); const sids: string[] = []; @@ -13,21 +13,21 @@ afterAll(async () => { await store.close(); }); -describe("SessionManager.openFromCheckpoint", () => { - it("loads only the tail slice from a checkpoint marker", async () => { +describe('SessionManager.openFromCheckpoint', () => { + it('loads only the tail slice from a checkpoint marker', async () => { const backend = new BufferedRedisBackend(store); const sm = SessionManager.create(process.cwd(), undefined, undefined, backend); const sid = sm.getSessionId(); sids.push(sid); - sm.appendMessage({ role: "user", content: "one" } as never); // ~pos 2 - const keepId = sm.appendMessage({ role: "assistant", content: "two" } as never); // ~pos 3 - sm.appendMessage({ role: "user", content: "three" } as never); // ~pos 4 + sm.appendMessage({ role: 'user', content: 'one' } as never); // ~pos 2 + const keepId = sm.appendMessage({ role: 'assistant', content: 'two' } as never); // ~pos 3 + sm.appendMessage({ role: 'user', content: 'three' } as never); // ~pos 4 await backend.flush(); const resumeFrom = await store.positionOfId(sid, keepId); expect(resumeFrom).not.toBeNull(); - sm.appendCustomEntry("checkpoint", { resumeFromPosition: resumeFrom }); // marker + sm.appendCustomEntry('checkpoint', { resumeFromPosition: resumeFrom }); // marker await backend.flush(); const resumed = await SessionManager.openFromCheckpoint(sid, backend, process.cwd()); @@ -41,13 +41,13 @@ describe("SessionManager.openFromCheckpoint", () => { expect(tail.length).toBeLessThan(full.length); }); - it("falls back to full reconstruction when there is no marker", async () => { + it('falls back to full reconstruction when there is no marker', async () => { const backend = new BufferedRedisBackend(store); const sm = SessionManager.create(process.cwd(), undefined, undefined, backend); const sid = sm.getSessionId(); sids.push(sid); - sm.appendMessage({ role: "user", content: "hi" } as never); - sm.appendMessage({ role: "assistant", content: "hello" } as never); + sm.appendMessage({ role: 'user', content: 'hi' } as never); + sm.appendMessage({ role: 'assistant', content: 'hello' } as never); await backend.flush(); const viaCheckpoint = await SessionManager.openFromCheckpoint(sid, backend, process.cwd()); @@ -58,21 +58,25 @@ describe("SessionManager.openFromCheckpoint", () => { }); }); -describe("checkpointExtension", () => { +describe('checkpointExtension', () => { it("writes a checkpoint marker pointing at the compaction's firstKeptEntryId on session_compact", async () => { const backend = new BufferedRedisBackend(store); const sm = SessionManager.create(process.cwd(), undefined, undefined, backend); const sid = sm.getSessionId(); sids.push(sid); - sm.appendMessage({ role: "user", content: "q" } as never); - const firstKeptId = sm.appendMessage({ role: "assistant", content: "a" } as never); - sm.appendCompaction("summary so far", firstKeptId, 1234); + sm.appendMessage({ role: 'user', content: 'q' } as never); + const firstKeptId = sm.appendMessage({ role: 'assistant', content: 'a' } as never); + sm.appendCompaction('summary so far', firstKeptId, 1234); await backend.flush(); // Capture the session_compact handler the extension registers. const handlers: Record = {}; - const pi = { on: (ev: string, h: Function) => { handlers[ev] = h; } }; + const pi = { + on: (ev: string, h: Function) => { + handlers[ev] = h; + }, + }; checkpointExtension(store, sm)(pi as never); await handlers.session_compact({ compactionEntry: { firstKeptEntryId: firstKeptId } }); @@ -80,28 +84,34 @@ describe("checkpointExtension", () => { const marker = await backend.latestCheckpoint(sid); const expectedPos = await store.positionOfId(sid, firstKeptId); - expect((marker as { customType?: string })?.customType).toBe("checkpoint"); - expect((marker as { data?: { resumeFromPosition?: number } })?.data?.resumeFromPosition).toBe(expectedPos); + expect((marker as { customType?: string })?.customType).toBe('checkpoint'); + expect((marker as { data?: { resumeFromPosition?: number } })?.data?.resumeFromPosition).toBe( + expectedPos, + ); }); }); -describe("reconstruction parity (M5 gate)", () => { - it("openFromCheckpoint buildSessionContext deep-equals openFromBackend after a compaction", async () => { +describe('reconstruction parity (M5 gate)', () => { + it('openFromCheckpoint buildSessionContext deep-equals openFromBackend after a compaction', async () => { const backend = new BufferedRedisBackend(store); const sm = SessionManager.create(process.cwd(), undefined, undefined, backend); const sid = sm.getSessionId(); sids.push(sid); - sm.appendMessage({ role: "user", content: "old turn 1" } as never); - sm.appendMessage({ role: "assistant", content: "old answer 1" } as never); - const firstKeptId = sm.appendMessage({ role: "user", content: "kept question" } as never); - sm.appendMessage({ role: "assistant", content: "kept answer" } as never); - sm.appendCompaction("summary of earlier turns", firstKeptId, 4321); + sm.appendMessage({ role: 'user', content: 'old turn 1' } as never); + sm.appendMessage({ role: 'assistant', content: 'old answer 1' } as never); + const firstKeptId = sm.appendMessage({ role: 'user', content: 'kept question' } as never); + sm.appendMessage({ role: 'assistant', content: 'kept answer' } as never); + sm.appendCompaction('summary of earlier turns', firstKeptId, 4321); await backend.flush(); // Run the real extension to write the marker. const handlers: Record = {}; - const pi = { on: (ev: string, h: Function) => { handlers[ev] = h; } }; + const pi = { + on: (ev: string, h: Function) => { + handlers[ev] = h; + }, + }; checkpointExtension(store, sm)(pi as never); await handlers.session_compact({ compactionEntry: { firstKeptEntryId: firstKeptId } }); await backend.flush(); @@ -112,36 +122,41 @@ describe("reconstruction parity (M5 gate)", () => { // And it really read a smaller slice than the full log. const cpMarker = await backend.latestCheckpoint(sid); - const resumeFrom = (cpMarker as { data?: { resumeFromPosition?: number } }).data!.resumeFromPosition!; + const resumeFrom = (cpMarker as { data?: { resumeFromPosition?: number } }).data! + .resumeFromPosition!; const tail = await store.read(sid, resumeFrom); const full = await store.read(sid); expect(tail.length).toBeLessThan(full.length); }); }); -describe("openFromCheckpoint thinkingLevel reset (documented tail-load caveat)", () => { - it("drops a pre-tail thinking_level_change: tail-load thinkingLevel is default, full-load is preserved", async () => { +describe('openFromCheckpoint thinkingLevel reset (documented tail-load caveat)', () => { + it('drops a pre-tail thinking_level_change: tail-load thinkingLevel is default, full-load is preserved', async () => { const backend = new BufferedRedisBackend(store); const sm = SessionManager.create(process.cwd(), undefined, undefined, backend); const sid = sm.getSessionId(); sids.push(sid); - sm.appendThinkingLevelChange("medium"); // pre-tail (before firstKept) - sm.appendMessage({ role: "user", content: "q" } as never); - const firstKeptId = sm.appendMessage({ role: "assistant", content: "a" } as never); - sm.appendMessage({ role: "user", content: "q2" } as never); - sm.appendCompaction("summary", firstKeptId, 1); + sm.appendThinkingLevelChange('medium'); // pre-tail (before firstKept) + sm.appendMessage({ role: 'user', content: 'q' } as never); + const firstKeptId = sm.appendMessage({ role: 'assistant', content: 'a' } as never); + sm.appendMessage({ role: 'user', content: 'q2' } as never); + sm.appendCompaction('summary', firstKeptId, 1); await backend.flush(); const handlers: Record = {}; - const pi = { on: (ev: string, h: Function) => { handlers[ev] = h; } }; + const pi = { + on: (ev: string, h: Function) => { + handlers[ev] = h; + }, + }; checkpointExtension(store, sm)(pi as never); await handlers.session_compact({ compactionEntry: { firstKeptEntryId: firstKeptId } }); await backend.flush(); const viaBackend = await SessionManager.openFromBackend(sid, backend, process.cwd()); const viaCheckpoint = await SessionManager.openFromCheckpoint(sid, backend, process.cwd()); - expect(viaBackend.buildSessionContext().thinkingLevel).toBe("medium"); // full load preserves - expect(viaCheckpoint.buildSessionContext().thinkingLevel).toBe("off"); // tail load resets (documented) + expect(viaBackend.buildSessionContext().thinkingLevel).toBe('medium'); // full load preserves + expect(viaCheckpoint.buildSessionContext().thinkingLevel).toBe('off'); // tail load resets (documented) }); }); diff --git a/harness/test/classify-outcome.test.ts b/harness/test/classify-outcome.test.ts index 98536a2..44d8329 100644 --- a/harness/test/classify-outcome.test.ts +++ b/harness/test/classify-outcome.test.ts @@ -1,26 +1,41 @@ -import { describe, it, expect } from "vitest"; -import { classifyOutcome } from "../src/classify-outcome"; -import type { LeafResult } from "../src/run-leaf"; +import { describe, it, expect } from 'vitest'; +import { classifyOutcome } from '../src/classify-outcome'; +import type { LeafResult } from '../src/run-leaf'; -describe("classifyOutcome", () => { - it("acks a done result (non-retryable)", () => { - expect(classifyOutcome({ status: "done", verdict: { item_id: "i", verdict: "CLEAR", reason: "r" } })) - .toEqual({ ack: true, retryable: false }); +describe('classifyOutcome', () => { + it('acks a done result (non-retryable)', () => { + expect( + classifyOutcome({ status: 'done', verdict: { item_id: 'i', verdict: 'CLEAR', reason: 'r' } }), + ).toEqual({ ack: true, retryable: false }); }); - it("acks a paused result (resume is a fresh invocation)", () => { - expect(classifyOutcome({ status: "paused", gateId: 1, gate: { summary: "s", proposed_action: "a" } })) - .toEqual({ ack: true, retryable: false }); + it('acks a paused result (resume is a fresh invocation)', () => { + expect( + classifyOutcome({ + status: 'paused', + gateId: 1, + gate: { summary: 's', proposed_action: 'a' }, + }), + ).toEqual({ ack: true, retryable: false }); }); - it("acks an aborted result", () => { - expect(classifyOutcome({ status: "aborted" })).toEqual({ ack: true, retryable: false }); + it('acks an aborted result', () => { + expect(classifyOutcome({ status: 'aborted' })).toEqual({ ack: true, retryable: false }); }); - it("acks a deterministic failure", () => { - expect(classifyOutcome({ status: "failed", reason: "no_verdict" })).toEqual({ ack: true, retryable: false }); + it('acks a deterministic failure', () => { + expect(classifyOutcome({ status: 'failed', reason: 'no_verdict' })).toEqual({ + ack: true, + retryable: false, + }); }); - it("retries a transient error without acking", () => { - expect(classifyOutcome({ status: "failed", reason: "error" })).toEqual({ ack: false, retryable: true }); + it('retries a transient error without acking', () => { + expect(classifyOutcome({ status: 'failed', reason: 'error' })).toEqual({ + ack: false, + retryable: true, + }); }); - it("retries a saturated result without acking (async drains as leases free, spec §4.3)", () => { - expect(classifyOutcome({ status: "failed", reason: "saturated" })).toEqual({ ack: false, retryable: true }); + it('retries a saturated result without acking (async drains as leases free, spec §4.3)', () => { + expect(classifyOutcome({ status: 'failed', reason: 'saturated' })).toEqual({ + ack: false, + retryable: true, + }); }); }); diff --git a/harness/test/converge.test.ts b/harness/test/converge.test.ts index ccfe834..68b2b18 100644 --- a/harness/test/converge.test.ts +++ b/harness/test/converge.test.ts @@ -1,143 +1,161 @@ -import { describe, it, expect } from "vitest"; +import { describe, it, expect } from 'vitest'; import { - leafWorkspaceRef, buildConvergeScript, buildCleanupScript, convergeWorkspace, - buildDiffCaptureScript, captureWorkspaceDiff, -} from "../src/converge.js"; + leafWorkspaceRef, + buildConvergeScript, + buildCleanupScript, + convergeWorkspace, + buildDiffCaptureScript, + captureWorkspaceDiff, +} from '../src/converge.js'; -describe("leafWorkspaceRef", () => { - it("is /workspace/leaves/", () => { - expect(leafWorkspaceRef("run-a-item-1")).toBe("/workspace/leaves/run-a-item-1"); +describe('leafWorkspaceRef', () => { + it('is /workspace/leaves/', () => { + expect(leafWorkspaceRef('run-a-item-1')).toBe('/workspace/leaves/run-a-item-1'); }); }); -describe("buildConvergeScript", () => { - const s = buildConvergeScript("https://git.example/r.git", "abc123", "leaf-1"); +describe('buildConvergeScript', () => { + const s = buildConvergeScript('https://git.example/r.git', 'abc123', 'leaf-1'); it("fetches the per-leaf repoUrl+ref explicitly (never a fixed 'origin')", () => { // #67: fetch must target the URL from this leaf's envelope, so one pooled // sandbox can serve many repos. It must NOT fetch a fixed `origin`. expect(s).toContain("fetch --quiet 'https://git.example/r.git' 'abc123'"); - expect(s).not.toContain("fetch --quiet origin"); + expect(s).not.toContain('fetch --quiet origin'); // No `git clone` — init+fetch replaces it (clone binds origin to the first URL). - expect(s).not.toContain("git clone"); + expect(s).not.toContain('git clone'); }); - it("does init+fetch inside the flocked subshell (no clone race)", () => { + it('does init+fetch inside the flocked subshell (no clone race)', () => { // #67 defect 2: the whole init+fetch must run under the flock, in order: // flock → init → fetch → close the lock fd. expect(s).toMatch(/flock 9[\s\S]*git init[\s\S]*fetch[\s\S]*9>"\$LOCK"/); }); - it("self-heals a missing or corrupt repo (init under lock, retry on fetch failure)", () => { + it('self-heals a missing or corrupt repo (init under lock, retry on fetch failure)', () => { // Missing/non-git /workspace/repo → rm -rf + git init (closes #59). expect(s).toContain('[ -d "$REPO/.git" ] || { rm -rf "$REPO"; git init'); // A failed fetch (e.g. corrupt .git) re-inits and fetches once more. expect(s).toMatch(/fetch --quiet '[^']*' '[^']*' \|\| \{ rm -rf "\$REPO"; git init/); }); - it("adds a per-leaf worktree at the fetched commit and prints the path", () => { - expect(s).toContain("worktree add"); - expect(s).toContain("/workspace/leaves/leaf-1"); + it('adds a per-leaf worktree at the fetched commit and prints the path', () => { + expect(s).toContain('worktree add'); + expect(s).toContain('/workspace/leaves/leaf-1'); expect(s).toContain('printf'); }); - it("single-quote-escapes inputs to resist injection", () => { - const evil = buildConvergeScript("https://x/r.git'; rm -rf /; '", "main", "leaf-1"); + it('single-quote-escapes inputs to resist injection', () => { + const evil = buildConvergeScript("https://x/r.git'; rm -rf /; '", 'main', 'leaf-1'); expect(evil).toContain(`'https://x/r.git'\\''; rm -rf /; '\\'''`); }); }); -describe("convergeWorkspace", () => { - it("returns trimmed stdout as the workspace ref on success", async () => { +describe('convergeWorkspace', () => { + it('returns trimmed stdout as the workspace ref on success', async () => { const transport = { - exec: async () => ({ stdout: Buffer.from("/workspace/leaves/leaf-1\n"), exitCode: 0, truncated: false }), + exec: async () => ({ + stdout: Buffer.from('/workspace/leaves/leaf-1\n'), + exitCode: 0, + truncated: false, + }), close: async () => {}, }; - expect(await convergeWorkspace(transport, "u", "r", "leaf-1")).toBe("/workspace/leaves/leaf-1"); + expect(await convergeWorkspace(transport, 'u', 'r', 'leaf-1')).toBe('/workspace/leaves/leaf-1'); }); - it("throws on non-zero exit", async () => { + it('throws on non-zero exit', async () => { const transport = { - exec: async () => ({ stdout: Buffer.from(""), exitCode: 1, truncated: false }), + exec: async () => ({ stdout: Buffer.from(''), exitCode: 1, truncated: false }), close: async () => {}, }; - await expect(convergeWorkspace(transport, "u", "r", "leaf-1")).rejects.toThrow(/converge failed/); + await expect(convergeWorkspace(transport, 'u', 'r', 'leaf-1')).rejects.toThrow( + /converge failed/, + ); }); - it("reports a capped converge as truncation, not a failed converge", async () => { + it('reports a capped converge as truncation, not a failed converge', async () => { // A converge whose fetch/worktree output overruns the sandbox output cap currently // surfaces as "converge failed (exit null)", which reads as a broken git command // rather than output too large for the seam. const transport = { - exec: async () => ({ stdout: Buffer.from("partial"), exitCode: null, truncated: true }), + exec: async () => ({ stdout: Buffer.from('partial'), exitCode: null, truncated: true }), close: async () => {}, }; - await expect(convergeWorkspace(transport, "u", "r", "leaf-1")).rejects.toThrow(/output cap/); + await expect(convergeWorkspace(transport, 'u', 'r', 'leaf-1')).rejects.toThrow(/output cap/); }); }); -describe("buildCleanupScript", () => { - it("removes the leaf worktree and prunes", () => { - const c = buildCleanupScript("leaf-1"); - expect(c).toContain("worktree remove"); - expect(c).toContain("/workspace/leaves/leaf-1"); - expect(c).toContain("worktree prune"); +describe('buildCleanupScript', () => { + it('removes the leaf worktree and prunes', () => { + const c = buildCleanupScript('leaf-1'); + expect(c).toContain('worktree remove'); + expect(c).toContain('/workspace/leaves/leaf-1'); + expect(c).toContain('worktree prune'); }); }); -describe("buildDiffCaptureScript", () => { - it("stages all edits then emits the cached diff, scoped to the leaf worktree", () => { - const s = buildDiffCaptureScript("run-1"); - expect(s).toContain("/workspace/leaves/run-1"); +describe('buildDiffCaptureScript', () => { + it('stages all edits then emits the cached diff, scoped to the leaf worktree', () => { + const s = buildDiffCaptureScript('run-1'); + expect(s).toContain('/workspace/leaves/run-1'); expect(s).toContain('git -C "$LEAF" add -A'); expect(s).toContain('git -C "$LEAF" diff --cached'); }); }); -describe("captureWorkspaceDiff", () => { - it("returns stdout as the patch on exit 0", async () => { +describe('captureWorkspaceDiff', () => { + it('returns stdout as the patch on exit 0', async () => { const transport = { - exec: async () => ({ stdout: Buffer.from("diff --git a/x b/x\n"), exitCode: 0, truncated: false }), + exec: async () => ({ + stdout: Buffer.from('diff --git a/x b/x\n'), + exitCode: 0, + truncated: false, + }), close: async () => {}, }; - expect(await captureWorkspaceDiff(transport, "run-1")).toBe("diff --git a/x b/x\n"); + expect(await captureWorkspaceDiff(transport, 'run-1')).toBe('diff --git a/x b/x\n'); }); - it("throws on non-zero exit", async () => { + it('throws on non-zero exit', async () => { const transport = { - exec: async () => ({ stdout: Buffer.from(""), exitCode: 3, truncated: false }), + exec: async () => ({ stdout: Buffer.from(''), exitCode: 3, truncated: false }), close: async () => {}, }; - await expect(captureWorkspaceDiff(transport, "run-1")).rejects.toThrow(/exit 3/); + await expect(captureWorkspaceDiff(transport, 'run-1')).rejects.toThrow(/exit 3/); }); - it("returns an empty string when the worktree has no changes (exit 0, empty stdout)", async () => { + it('returns an empty string when the worktree has no changes (exit 0, empty stdout)', async () => { const transport = { - exec: async () => ({ stdout: Buffer.from(""), exitCode: 0, truncated: false }), + exec: async () => ({ stdout: Buffer.from(''), exitCode: 0, truncated: false }), close: async () => {}, }; - expect(await captureWorkspaceDiff(transport, "run-1")).toBe(""); + expect(await captureWorkspaceDiff(transport, 'run-1')).toBe(''); }); - it("restores a trailing newline the transport stripped (patch must not end mid-line)", async () => { + it('restores a trailing newline the transport stripped (patch must not end mid-line)', async () => { // Some exec transports drop the trailing newline; a diff that ends mid-line is rejected by // `git apply` / GNU patch, so captureWorkspaceDiff must normalize it back. const transport = { exec: async () => ({ - stdout: Buffer.from("diff --git a/x b/x\n@@ -1 +1 @@\n-a\n+b"), + stdout: Buffer.from('diff --git a/x b/x\n@@ -1 +1 @@\n-a\n+b'), exitCode: 0, truncated: false, }), close: async () => {}, }; - const patch = await captureWorkspaceDiff(transport, "run-1"); - expect(patch.endsWith("\n")).toBe(true); - expect(patch).toBe("diff --git a/x b/x\n@@ -1 +1 @@\n-a\n+b\n"); + const patch = await captureWorkspaceDiff(transport, 'run-1'); + expect(patch.endsWith('\n')).toBe(true); + expect(patch).toBe('diff --git a/x b/x\n@@ -1 +1 @@\n-a\n+b\n'); }); - it("does not add a second newline when the patch already ends with one", async () => { + it('does not add a second newline when the patch already ends with one', async () => { const transport = { - exec: async () => ({ stdout: Buffer.from("diff --git a/x b/x\n"), exitCode: 0, truncated: false }), + exec: async () => ({ + stdout: Buffer.from('diff --git a/x b/x\n'), + exitCode: 0, + truncated: false, + }), close: async () => {}, }; - expect(await captureWorkspaceDiff(transport, "run-1")).toBe("diff --git a/x b/x\n"); + expect(await captureWorkspaceDiff(transport, 'run-1')).toBe('diff --git a/x b/x\n'); }); - it("reports a capped diff as truncation, not a failed capture", async () => { + it('reports a capped diff as truncation, not a failed capture', async () => { // A >8 MiB diff currently surfaces as "diff capture failed (exit null)", which reads // as a broken git command rather than a diff too large for the seam. const transport = { - exec: async () => ({ stdout: Buffer.from("partial"), exitCode: null, truncated: true }), + exec: async () => ({ stdout: Buffer.from('partial'), exitCode: null, truncated: true }), close: async () => {}, }; - await expect(captureWorkspaceDiff(transport, "run-1")).rejects.toThrow(/output cap/); + await expect(captureWorkspaceDiff(transport, 'run-1')).rejects.toThrow(/output cap/); }); }); diff --git a/harness/test/fixtures.test.ts b/harness/test/fixtures.test.ts index e545be1..ce76f03 100644 --- a/harness/test/fixtures.test.ts +++ b/harness/test/fixtures.test.ts @@ -1,19 +1,19 @@ -import { describe, it, expect } from "vitest"; -import { readFileSync, existsSync } from "node:fs"; -import { join } from "node:path"; -import { fileURLToPath } from "node:url"; -import { dirname } from "node:path"; +import { describe, it, expect } from 'vitest'; +import { readFileSync, existsSync } from 'node:fs'; +import { join } from 'node:path'; +import { fileURLToPath } from 'node:url'; +import { dirname } from 'node:path'; const __dirname = dirname(fileURLToPath(import.meta.url)); -const root = join(__dirname, "../../deploy/knative/fixtures"); +const root = join(__dirname, '../../deploy/knative/fixtures'); -describe("fixtures", () => { - for (const id of ["i1", "i2", "i3"]) { +describe('fixtures', () => { + for (const id of ['i1', 'i2', 'i3']) { it(`${id}.json references an existing fixture file`, () => { - const item = JSON.parse(readFileSync(join(root, "inputs", `${id}.json`), "utf8")); - expect(typeof item.item_id).toBe("string"); - expect(typeof item.pattern).toBe("string"); - expect(existsSync(join(root, "repo", item.file))).toBe(true); + const item = JSON.parse(readFileSync(join(root, 'inputs', `${id}.json`), 'utf8')); + expect(typeof item.item_id).toBe('string'); + expect(typeof item.pattern).toBe('string'); + expect(existsSync(join(root, 'repo', item.file))).toBe(true); }); } }); diff --git a/harness/test/gate.test.ts b/harness/test/gate.test.ts index 6c50d28..d1a8cf0 100644 --- a/harness/test/gate.test.ts +++ b/harness/test/gate.test.ts @@ -1,142 +1,190 @@ -import { describe, it, expect, beforeEach, afterEach } from "vitest"; -import { mkdtempSync, writeFileSync, rmSync } from "node:fs"; -import { join } from "node:path"; -import { tmpdir } from "node:os"; +import { describe, it, expect, beforeEach, afterEach } from 'vitest'; +import { mkdtempSync, writeFileSync, rmSync } from 'node:fs'; +import { join } from 'node:path'; +import { tmpdir } from 'node:os'; import { validateDecision, - isGateRequestEntry, isGateDecisionEntry, gateRequestFromEntry, gateDecisionFromEntry, - GATE_REQUEST_ENTRY_TYPE, GATE_DECISION_ENTRY_TYPE, -} from "../src/gate"; + isGateRequestEntry, + isGateDecisionEntry, + gateRequestFromEntry, + gateDecisionFromEntry, + GATE_REQUEST_ENTRY_TYPE, + GATE_DECISION_ENTRY_TYPE, +} from '../src/gate'; let dir: string; -beforeEach(() => { dir = mkdtempSync(join(tmpdir(), "gate-")); }); -afterEach(() => { rmSync(dir, { recursive: true, force: true }); }); +beforeEach(() => { + dir = mkdtempSync(join(tmpdir(), 'gate-')); +}); +afterEach(() => { + rmSync(dir, { recursive: true, force: true }); +}); -describe("validateDecision", () => { - it("accepts approve/reject/abort with a numeric gateId", () => { - expect(validateDecision({ gateId: 0, action: "approve" })).toEqual({ - ok: true, value: { gateId: 0, action: "approve", feedback: undefined }, +describe('validateDecision', () => { + it('accepts approve/reject/abort with a numeric gateId', () => { + expect(validateDecision({ gateId: 0, action: 'approve' })).toEqual({ + ok: true, + value: { gateId: 0, action: 'approve', feedback: undefined }, }); - expect(validateDecision({ gateId: 2, action: "reject", feedback: "do X" })).toEqual({ - ok: true, value: { gateId: 2, action: "reject", feedback: "do X" }, + expect(validateDecision({ gateId: 2, action: 'reject', feedback: 'do X' })).toEqual({ + ok: true, + value: { gateId: 2, action: 'reject', feedback: 'do X' }, }); - expect(validateDecision({ gateId: 1, action: "abort" }).ok).toBe(true); + expect(validateDecision({ gateId: 1, action: 'abort' }).ok).toBe(true); }); - it("rejects a bad action, missing gateId, or non-object", () => { - expect(validateDecision({ gateId: 0, action: "maybe" }).ok).toBe(false); - expect(validateDecision({ action: "approve" }).ok).toBe(false); + it('rejects a bad action, missing gateId, or non-object', () => { + expect(validateDecision({ gateId: 0, action: 'maybe' }).ok).toBe(false); + expect(validateDecision({ action: 'approve' }).ok).toBe(false); expect(validateDecision(null).ok).toBe(false); }); }); -describe("entry helpers", () => { - const req = { type: "custom", customType: GATE_REQUEST_ENTRY_TYPE, data: { gateId: 0, summary: "s", proposed_action: "a" } }; - const dec = { type: "custom", customType: GATE_DECISION_ENTRY_TYPE, data: { gateId: 0, action: "approve" } }; - it("recognizes and extracts gate-request entries", () => { +describe('entry helpers', () => { + const req = { + type: 'custom', + customType: GATE_REQUEST_ENTRY_TYPE, + data: { gateId: 0, summary: 's', proposed_action: 'a' }, + }; + const dec = { + type: 'custom', + customType: GATE_DECISION_ENTRY_TYPE, + data: { gateId: 0, action: 'approve' }, + }; + it('recognizes and extracts gate-request entries', () => { expect(isGateRequestEntry(req)).toBe(true); expect(isGateRequestEntry(dec)).toBe(false); - expect(gateRequestFromEntry(req)).toEqual({ gateId: 0, summary: "s", proposed_action: "a" }); + expect(gateRequestFromEntry(req)).toEqual({ gateId: 0, summary: 's', proposed_action: 'a' }); expect(gateRequestFromEntry(dec)).toBeNull(); }); - it("recognizes and extracts gate-decision entries", () => { + it('recognizes and extracts gate-decision entries', () => { expect(isGateDecisionEntry(dec)).toBe(true); - expect(gateDecisionFromEntry(dec)).toEqual({ gateId: 0, action: "approve", feedback: undefined }); + expect(gateDecisionFromEntry(dec)).toEqual({ + gateId: 0, + action: 'approve', + feedback: undefined, + }); expect(gateDecisionFromEntry(req)).toBeNull(); }); }); -import { computeGateState, continuationPrompt } from "../src/gate"; +import { computeGateState, continuationPrompt } from '../src/gate'; function reqEntry(gateId: number) { - return { type: "custom", customType: "gate-request", data: { gateId, summary: `s${gateId}`, proposed_action: `a${gateId}` } }; + return { + type: 'custom', + customType: 'gate-request', + data: { gateId, summary: `s${gateId}`, proposed_action: `a${gateId}` }, + }; } -function decEntry(gateId: number, action = "approve", feedback?: string) { - return { type: "custom", customType: "gate-decision", data: { gateId, action, feedback } }; +function decEntry(gateId: number, action = 'approve', feedback?: string) { + return { type: 'custom', customType: 'gate-decision', data: { gateId, action, feedback } }; } -describe("computeGateState", () => { - it("no gates → no pending, nextGateId 0", () => { - const s = computeGateState([{ type: "user" }]); +describe('computeGateState', () => { + it('no gates → no pending, nextGateId 0', () => { + const s = computeGateState([{ type: 'user' }]); expect(s.pendingGate).toBeNull(); expect(s.lastDecision).toBeNull(); expect(s.nextGateId).toBe(0); }); - it("one undecided request → it is pending, nextGateId 1", () => { + it('one undecided request → it is pending, nextGateId 1', () => { const s = computeGateState([reqEntry(0)]); - expect(s.pendingGate).toEqual({ gateId: 0, summary: "s0", proposed_action: "a0" }); + expect(s.pendingGate).toEqual({ gateId: 0, summary: 's0', proposed_action: 'a0' }); expect(s.nextGateId).toBe(1); }); - it("decided request → no pending, lastDecision set", () => { - const s = computeGateState([reqEntry(0), decEntry(0, "approve")]); + it('decided request → no pending, lastDecision set', () => { + const s = computeGateState([reqEntry(0), decEntry(0, 'approve')]); expect(s.pendingGate).toBeNull(); - expect(s.lastDecision).toEqual({ gateId: 0, action: "approve", feedback: undefined }); + expect(s.lastDecision).toEqual({ gateId: 0, action: 'approve', feedback: undefined }); }); - it("second request after a decided first → second is pending, nextGateId 2", () => { + it('second request after a decided first → second is pending, nextGateId 2', () => { const s = computeGateState([reqEntry(0), decEntry(0), reqEntry(1)]); expect(s.pendingGate?.gateId).toBe(1); expect(s.nextGateId).toBe(2); }); }); -describe("continuationPrompt", () => { - it("approve mentions APPROVED and submit_verdict", () => { - const p = continuationPrompt("approve", "looks good"); - expect(p).toContain("APPROVED"); - expect(p).toContain("looks good"); - expect(p).toContain("submit_verdict"); - }); - it("reject mentions REJECTED and revise", () => { - const p = continuationPrompt("reject", "fix the query"); - expect(p).toContain("REJECTED"); - expect(p).toContain("fix the query"); - expect(p.toLowerCase()).toContain("revise"); +describe('continuationPrompt', () => { + it('approve mentions APPROVED and submit_verdict', () => { + const p = continuationPrompt('approve', 'looks good'); + expect(p).toContain('APPROVED'); + expect(p).toContain('looks good'); + expect(p).toContain('submit_verdict'); + }); + it('reject mentions REJECTED and revise', () => { + const p = continuationPrompt('reject', 'fix the query'); + expect(p).toContain('REJECTED'); + expect(p).toContain('fix the query'); + expect(p.toLowerCase()).toContain('revise'); }); }); -import { decideSeed } from "../src/gate"; +import { decideSeed } from '../src/gate'; -const FRESH = "FRESH_PROMPT"; -function state(entries: unknown[]) { return computeGateState(entries); } +const FRESH = 'FRESH_PROMPT'; +function state(entries: unknown[]) { + return computeGateState(entries); +} -describe("decideSeed", () => { - it("fresh (no gates) → seed the fresh prompt, no record", () => { - expect(decideSeed(state([]), null, FRESH)).toEqual({ kind: "seed", prompt: FRESH, record: null }); - }); - it("pending gate + matching approve → seed continuation + record decision", () => { - const r = decideSeed(state([reqEntry(0)]), { gateId: 0, action: "approve" }, FRESH); - expect(r.kind).toBe("seed"); - if (r.kind === "seed") { - expect(r.prompt).toContain("APPROVED"); - expect(r.record).toEqual({ gateId: 0, action: "approve", feedback: undefined }); - } +describe('decideSeed', () => { + it('fresh (no gates) → seed the fresh prompt, no record', () => { + expect(decideSeed(state([]), null, FRESH)).toEqual({ + kind: 'seed', + prompt: FRESH, + record: null, + }); }); - it("pending gate + matching reject → seed continuation with feedback", () => { - const r = decideSeed(state([reqEntry(0)]), { gateId: 0, action: "reject", feedback: "redo" }, FRESH); - expect(r.kind).toBe("seed"); - if (r.kind === "seed") expect(r.prompt).toContain("redo"); + it('pending gate + matching approve → seed continuation + record decision', () => { + const r = decideSeed(state([reqEntry(0)]), { gateId: 0, action: 'approve' }, FRESH); + expect(r.kind).toBe('seed'); + if (r.kind === 'seed') { + expect(r.prompt).toContain('APPROVED'); + expect(r.record).toEqual({ gateId: 0, action: 'approve', feedback: undefined }); + } }); - it("pending gate + matching abort → abort + record", () => { - expect(decideSeed(state([reqEntry(0)]), { gateId: 0, action: "abort" }, FRESH)).toEqual({ - kind: "abort", record: { gateId: 0, action: "abort", feedback: undefined }, + it('pending gate + matching reject → seed continuation with feedback', () => { + const r = decideSeed( + state([reqEntry(0)]), + { gateId: 0, action: 'reject', feedback: 'redo' }, + FRESH, + ); + expect(r.kind).toBe('seed'); + if (r.kind === 'seed') expect(r.prompt).toContain('redo'); + }); + it('pending gate + matching abort → abort + record', () => { + expect(decideSeed(state([reqEntry(0)]), { gateId: 0, action: 'abort' }, FRESH)).toEqual({ + kind: 'abort', + record: { gateId: 0, action: 'abort', feedback: undefined }, }); }); - it("pending gate + no decision → paused", () => { + it('pending gate + no decision → paused', () => { expect(decideSeed(state([reqEntry(0)]), null, FRESH)).toEqual({ - kind: "paused", gate: { gateId: 0, summary: "s0", proposed_action: "a0" }, + kind: 'paused', + gate: { gateId: 0, summary: 's0', proposed_action: 'a0' }, }); }); - it("pending gate + gateId mismatch → paused (stale decision ignored)", () => { - expect(decideSeed(state([reqEntry(1)]), { gateId: 0, action: "approve" }, FRESH).kind).toBe("paused"); + it('pending gate + gateId mismatch → paused (stale decision ignored)', () => { + expect(decideSeed(state([reqEntry(1)]), { gateId: 0, action: 'approve' }, FRESH).kind).toBe( + 'paused', + ); }); - it("already-decided this gate (double resume) → seed continuation, record null (no re-record)", () => { + it('already-decided this gate (double resume) → seed continuation, record null (no re-record)', () => { // The decision entry for gate 0 already exists, so the gate is no longer pending. - const r = decideSeed(state([reqEntry(0), decEntry(0, "approve")]), { gateId: 0, action: "approve" }, FRESH); - expect(r.kind).toBe("seed"); - if (r.kind === "seed") { expect(r.prompt).toContain("APPROVED"); expect(r.record).toBeNull(); } + const r = decideSeed( + state([reqEntry(0), decEntry(0, 'approve')]), + { gateId: 0, action: 'approve' }, + FRESH, + ); + expect(r.kind).toBe('seed'); + if (r.kind === 'seed') { + expect(r.prompt).toContain('APPROVED'); + expect(r.record).toBeNull(); + } }); - it("already-aborted session (no pending, last decision abort) → terminal abort, record null", () => { - expect(decideSeed(state([reqEntry(0), decEntry(0, "abort")]), null, FRESH)).toEqual({ - kind: "abort", record: null, + it('already-aborted session (no pending, last decision abort) → terminal abort, record null', () => { + expect(decideSeed(state([reqEntry(0), decEntry(0, 'abort')]), null, FRESH)).toEqual({ + kind: 'abort', + record: null, }); }); }); diff --git a/harness/test/integration.test.ts b/harness/test/integration.test.ts index 983f9f7..0678f1f 100644 --- a/harness/test/integration.test.ts +++ b/harness/test/integration.test.ts @@ -1,9 +1,9 @@ -import { describe, it, expect, afterAll } from "vitest"; -import { SessionManager, type FileEntry } from "@earendil-works/pi-coding-agent"; -import { RedisSessionBackend } from "@sh/session-backend"; -import { BufferedRedisBackend } from "../src/buffered-redis-backend"; +import { describe, it, expect, afterAll } from 'vitest'; +import { SessionManager, type FileEntry } from '@earendil-works/pi-coding-agent'; +import { RedisSessionBackend } from '@sh/session-backend'; +import { BufferedRedisBackend } from '../src/buffered-redis-backend'; -const REDIS = process.env.REDIS_URL ?? "redis://127.0.0.1:6379"; +const REDIS = process.env.REDIS_URL ?? 'redis://127.0.0.1:6379'; const store = new RedisSessionBackend(REDIS); let createdSid: string | undefined; @@ -12,17 +12,17 @@ afterAll(async () => { await store.close(); }); -describe("SessionManager + Redis (parity / mobility / recovery)", () => { - it("a completed turn survives process death and a fresh instance resumes it", async () => { +describe('SessionManager + Redis (parity / mobility / recovery)', () => { + it('a completed turn survives process death and a fresh instance resumes it', async () => { const backend = new BufferedRedisBackend(store); // Drive a "turn": user -> assistant -> checkpoint -> user. const sm = SessionManager.create(process.cwd(), undefined, undefined, backend); createdSid = sm.getSessionId(); - sm.appendMessage({ role: "user", content: "hello" } as never); - sm.appendMessage({ role: "assistant", content: "hi there" } as never); - sm.appendCustomEntry("checkpoint", { ctx: "reconstructed" }); - sm.appendMessage({ role: "user", content: "again" } as never); + sm.appendMessage({ role: 'user', content: 'hello' } as never); + sm.appendMessage({ role: 'assistant', content: 'hi there' } as never); + sm.appendCustomEntry('checkpoint', { ctx: 'reconstructed' }); + sm.appendMessage({ role: 'user', content: 'again' } as never); // Durability barrier (what the harness calls at turn_end). await backend.flush(); @@ -40,6 +40,6 @@ describe("SessionManager + Redis (parity / mobility / recovery)", () => { // Checkpoint is recoverable through the decorator. const cp = await backend.latestCheckpoint(createdSid); - expect((cp as { customType?: string })?.customType).toBe("checkpoint"); + expect((cp as { customType?: string })?.customType).toBe('checkpoint'); }); }); diff --git a/harness/test/leaf-job-runner.test.ts b/harness/test/leaf-job-runner.test.ts index b4470af..47f317b 100644 --- a/harness/test/leaf-job-runner.test.ts +++ b/harness/test/leaf-job-runner.test.ts @@ -1,19 +1,27 @@ // harness/test/leaf-job-runner.test.ts -import { describe, it, expect, vi } from "vitest"; -import { processOne, type LeafJobDeps } from "../src/leaf-job-runner"; -import type { ClaimedEntry, WorkQueue } from "@sh/work-queue"; -import type { LeafEnvelope } from "../src/run-leaf"; -import type { RedisLike, LeafResultRecord } from "../src/leaf-result-store"; +import { describe, it, expect, vi } from 'vitest'; +import { processOne, type LeafJobDeps } from '../src/leaf-job-runner'; +import type { ClaimedEntry, WorkQueue } from '@sh/work-queue'; +import type { LeafEnvelope } from '../src/run-leaf'; +import type { RedisLike, LeafResultRecord } from '../src/leaf-result-store'; -function fakeQueue(claimed: ClaimedEntry | null): WorkQueue & { acked: string[]; touched: string[] } { - const acked: string[] = []; const touched: string[] = []; +function fakeQueue( + claimed: ClaimedEntry | null, +): WorkQueue & { acked: string[]; touched: string[] } { + const acked: string[] = []; + const touched: string[] = []; return { - acked, touched, + acked, + touched, ensureGroup: async () => {}, - enqueue: async () => "1-0", + enqueue: async () => '1-0', claim: async () => claimed, - ack: async (id: string) => { acked.push(id); }, - touch: async (id: string) => { touched.push(id); }, + ack: async (id: string) => { + acked.push(id); + }, + touch: async (id: string) => { + touched.push(id); + }, pending: async () => 0, purge: async () => {}, close: async () => {}, @@ -22,116 +30,168 @@ function fakeQueue(claimed: ClaimedEntry | null): WorkQueue & { acked: string[]; function fakeStore() { const m = new Map(); - const store: RedisLike = { async set(k, v) { m.set(k, v); }, async get(k) { return m.get(k) ?? null; } }; - return { store, get: (id: string) => { const r = m.get(`leaf:result:${id}`); return r ? JSON.parse(r) as LeafResultRecord : null; } }; + const store: RedisLike = { + async set(k, v) { + m.set(k, v); + }, + async get(k) { + return m.get(k) ?? null; + }, + }; + return { + store, + get: (id: string) => { + const r = m.get(`leaf:result:${id}`); + return r ? (JSON.parse(r) as LeafResultRecord) : null; + }, + }; } // Inline envelope shape: sessionId "run/i1" → leafSessionId sanitizes slash→dash → "run-i1" -const ENV: LeafEnvelope = { sessionId: "run/i1", item: { item_id: "i1", file: "f", pattern: "p" } }; +const ENV: LeafEnvelope = { sessionId: 'run/i1', item: { item_id: 'i1', file: 'f', pattern: 'p' } }; function baseDeps(over: Partial): LeafJobDeps { const { store } = fakeStore(); return { queue: fakeQueue(null), - runLeaf: async () => ({ status: "done", verdict: { item_id: "i1", verdict: "CLEAR", reason: "ok" } }), + runLeaf: async () => ({ + status: 'done', + verdict: { item_id: 'i1', verdict: 'CLEAR', reason: 'ok' }, + }), resultStore: store, ttlSeconds: 3600, - consumerId: "w1", - now: () => "t", + consumerId: 'w1', + now: () => 't', setHeartbeat: () => 0, clearHeartbeat: () => {}, ...over, }; } -describe("processOne", () => { - it("returns idle and does nothing when the queue is empty", async () => { +describe('processOne', () => { + it('returns idle and does nothing when the queue is empty', async () => { const q = fakeQueue(null); const r = await processOne(baseDeps({ queue: q })); - expect(r).toBe("idle"); + expect(r).toBe('idle'); expect(q.acked).toEqual([]); }); - it("dead-letters (failed record + ack, runLeaf NOT called) past maxAttempts", async () => { - const q = fakeQueue({ entryId: "9-0", envelope: ENV, deliveryCount: 4 }); + it('dead-letters (failed record + ack, runLeaf NOT called) past maxAttempts', async () => { + const q = fakeQueue({ entryId: '9-0', envelope: ENV, deliveryCount: 4 }); const runLeaf = vi.fn(); const { store, get } = fakeStore(); - const r = await processOne(baseDeps({ queue: q, runLeaf: runLeaf as any, maxAttempts: 3, resultStore: store })); - expect(r).toBe("deadletter"); + const r = await processOne( + baseDeps({ queue: q, runLeaf: runLeaf as any, maxAttempts: 3, resultStore: store }), + ); + expect(r).toBe('deadletter'); expect(runLeaf).not.toHaveBeenCalled(); - expect(get("run-i1")).toMatchObject({ status: "failed", reason: "error" }); - expect(q.acked).toEqual(["9-0"]); + expect(get('run-i1')).toMatchObject({ status: 'failed', reason: 'error' }); + expect(q.acked).toEqual(['9-0']); }); - it("writes a done record and acks", async () => { - const q = fakeQueue({ entryId: "1-0", envelope: ENV, deliveryCount: 1 }); + it('writes a done record and acks', async () => { + const q = fakeQueue({ entryId: '1-0', envelope: ENV, deliveryCount: 1 }); const { store, get } = fakeStore(); - const outcome = await processOne(baseDeps({ - queue: q, - resultStore: store, - runLeaf: async () => ({ status: "done", verdict: { item_id: "i1", verdict: "FLAGGED", reason: "x" } }), - })); - expect(outcome).toBe("done"); - expect(get("run-i1")).toMatchObject({ status: "done", sessionId: "run/i1" }); - expect(q.acked).toEqual(["1-0"]); + const outcome = await processOne( + baseDeps({ + queue: q, + resultStore: store, + runLeaf: async () => ({ + status: 'done', + verdict: { item_id: 'i1', verdict: 'FLAGGED', reason: 'x' }, + }), + }), + ); + expect(outcome).toBe('done'); + expect(get('run-i1')).toMatchObject({ status: 'done', sessionId: 'run/i1' }); + expect(q.acked).toEqual(['1-0']); }); - it("deterministic failure → failed record + ack", async () => { - const q = fakeQueue({ entryId: "1-0", envelope: ENV, deliveryCount: 1 }); + it('deterministic failure → failed record + ack', async () => { + const q = fakeQueue({ entryId: '1-0', envelope: ENV, deliveryCount: 1 }); const { store, get } = fakeStore(); - const r = await processOne(baseDeps({ queue: q, resultStore: store, runLeaf: async () => ({ status: "failed", reason: "bad_inputs" }) })); - expect(r).toBe("failed"); - expect(get("run-i1")).toMatchObject({ status: "failed", reason: "bad_inputs" }); - expect(q.acked).toEqual(["1-0"]); + const r = await processOne( + baseDeps({ + queue: q, + resultStore: store, + runLeaf: async () => ({ status: 'failed', reason: 'bad_inputs' }), + }), + ); + expect(r).toBe('failed'); + expect(get('run-i1')).toMatchObject({ status: 'failed', reason: 'bad_inputs' }); + expect(q.acked).toEqual(['1-0']); }); - it("paused → acks, writes paused record, returns paused", async () => { - const q = fakeQueue({ entryId: "1-0", envelope: ENV, deliveryCount: 1 }); + it('paused → acks, writes paused record, returns paused', async () => { + const q = fakeQueue({ entryId: '1-0', envelope: ENV, deliveryCount: 1 }); const { store, get } = fakeStore(); - const r = await processOne(baseDeps({ queue: q, resultStore: store, runLeaf: async () => ({ status: "paused", gateId: 0, gate: { summary: "s", proposed_action: "a" } }) })); - expect(r).toBe("paused"); - expect(get("run-i1")).toMatchObject({ status: "paused" }); - expect(q.acked).toEqual(["1-0"]); + const r = await processOne( + baseDeps({ + queue: q, + resultStore: store, + runLeaf: async () => ({ + status: 'paused', + gateId: 0, + gate: { summary: 's', proposed_action: 'a' }, + }), + }), + ); + expect(r).toBe('paused'); + expect(get('run-i1')).toMatchObject({ status: 'paused' }); + expect(q.acked).toEqual(['1-0']); }); - it("aborted → writes aborted record + ack, returns aborted", async () => { - const q = fakeQueue({ entryId: "1-0", envelope: ENV, deliveryCount: 1 }); + it('aborted → writes aborted record + ack, returns aborted', async () => { + const q = fakeQueue({ entryId: '1-0', envelope: ENV, deliveryCount: 1 }); const { store, get } = fakeStore(); - const r = await processOne(baseDeps({ queue: q, resultStore: store, runLeaf: async () => ({ status: "aborted" }) })); - expect(r).toBe("aborted"); - expect(get("run-i1")).toMatchObject({ status: "aborted" }); - expect(q.acked).toEqual(["1-0"]); + const r = await processOne( + baseDeps({ queue: q, resultStore: store, runLeaf: async () => ({ status: 'aborted' }) }), + ); + expect(r).toBe('aborted'); + expect(get('run-i1')).toMatchObject({ status: 'aborted' }); + expect(q.acked).toEqual(['1-0']); }); - it("responded → writes a record + ack, returns responded", async () => { - const q = fakeQueue({ entryId: "1-0", envelope: ENV, deliveryCount: 1 }); + it('responded → writes a record + ack, returns responded', async () => { + const q = fakeQueue({ entryId: '1-0', envelope: ENV, deliveryCount: 1 }); const { store, get } = fakeStore(); - const r = await processOne(baseDeps({ queue: q, resultStore: store, runLeaf: async () => ({ status: "responded", text: "hi" }) })); - expect(r).toBe("responded"); + const r = await processOne( + baseDeps({ + queue: q, + resultStore: store, + runLeaf: async () => ({ status: 'responded', text: 'hi' }), + }), + ); + expect(r).toBe('responded'); // A result record is written and the entry is acked (a prompt leaf is a terminal success). // The persisted record's shape for "responded" is leaf-result-store's concern (a later task); // here we only assert the runner's ack + return behavior. - expect(get("run-i1")).not.toBeNull(); - expect(q.acked).toEqual(["1-0"]); + expect(get('run-i1')).not.toBeNull(); + expect(q.acked).toEqual(['1-0']); }); - it("does not write a record and does not ack on transient error (retry)", async () => { - const q = fakeQueue({ entryId: "1-0", envelope: ENV, deliveryCount: 1 }); + it('does not write a record and does not ack on transient error (retry)', async () => { + const q = fakeQueue({ entryId: '1-0', envelope: ENV, deliveryCount: 1 }); const { store, get } = fakeStore(); - const outcome = await processOne(baseDeps({ - queue: q, - resultStore: store, - runLeaf: async () => ({ status: "failed", reason: "error" }), - })); - expect(outcome).toBe("retry"); - expect(get("run-i1")).toBeNull(); + const outcome = await processOne( + baseDeps({ + queue: q, + resultStore: store, + runLeaf: async () => ({ status: 'failed', reason: 'error' }), + }), + ); + expect(outcome).toBe('retry'); + expect(get('run-i1')).toBeNull(); expect(q.acked).toEqual([]); }); - it("schedules and clears a heartbeat around the run", async () => { - const q = fakeQueue({ entryId: "1-0", envelope: ENV, deliveryCount: 1 }); - const set = vi.fn(() => 42); const clear = vi.fn(); - await processOne(baseDeps({ queue: q, setHeartbeat: set as any, clearHeartbeat: clear as any })); + it('schedules and clears a heartbeat around the run', async () => { + const q = fakeQueue({ entryId: '1-0', envelope: ENV, deliveryCount: 1 }); + const set = vi.fn(() => 42); + const clear = vi.fn(); + await processOne( + baseDeps({ queue: q, setHeartbeat: set as any, clearHeartbeat: clear as any }), + ); expect(set).toHaveBeenCalledOnce(); expect(clear).toHaveBeenCalledWith(42); }); diff --git a/harness/test/leaf-result-store.test.ts b/harness/test/leaf-result-store.test.ts index b16c4cf..38018a4 100644 --- a/harness/test/leaf-result-store.test.ts +++ b/harness/test/leaf-result-store.test.ts @@ -1,107 +1,162 @@ -import { describe, it, expect } from "vitest"; +import { describe, it, expect } from 'vitest'; import { - resultKey, toResultRecord, writeResult, readResult, type RedisLike, type LeafResultRecord, -} from "../src/leaf-result-store"; -import type { LeafResult } from "../src/run-leaf"; + resultKey, + toResultRecord, + writeResult, + readResult, + type RedisLike, + type LeafResultRecord, +} from '../src/leaf-result-store'; +import type { LeafResult } from '../src/run-leaf'; function fakeRedis(): RedisLike & { store: Map; ttl: Map } { const store = new Map(); const ttl = new Map(); return { - store, ttl, - async set(key, value, opts) { store.set(key, value); if (opts?.EX) ttl.set(key, opts.EX); }, - async get(key) { return store.get(key) ?? null; }, + store, + ttl, + async set(key, value, opts) { + store.set(key, value); + if (opts?.EX) ttl.set(key, opts.EX); + }, + async get(key) { + return store.get(key) ?? null; + }, }; } -describe("resultKey", () => { - it("namespaces by leaf session id", () => { - expect(resultKey("run-1-i1")).toBe("leaf:result:run-1-i1"); +describe('resultKey', () => { + it('namespaces by leaf session id', () => { + expect(resultKey('run-1-i1')).toBe('leaf:result:run-1-i1'); }); }); -describe("toResultRecord", () => { - it("maps done → verdict-bearing record", () => { - const r: LeafResult = { status: "done", verdict: { item_id: "i1", verdict: "FLAGGED", reason: "x" } }; - expect(toResultRecord(r, "run-1/i1", "T")).toEqual({ - status: "done", verdict: { item_id: "i1", verdict: "FLAGGED", reason: "x" }, - gate: null, reason: null, patch: null, text: null, usage: null, sessionId: "run-1/i1", ts: "T", +describe('toResultRecord', () => { + it('maps done → verdict-bearing record', () => { + const r: LeafResult = { + status: 'done', + verdict: { item_id: 'i1', verdict: 'FLAGGED', reason: 'x' }, + }; + expect(toResultRecord(r, 'run-1/i1', 'T')).toEqual({ + status: 'done', + verdict: { item_id: 'i1', verdict: 'FLAGGED', reason: 'x' }, + gate: null, + reason: null, + patch: null, + text: null, + usage: null, + sessionId: 'run-1/i1', + ts: 'T', }); }); - it("maps paused → gate-bearing record", () => { - const r: LeafResult = { status: "paused", gateId: 1, gate: { summary: "s", proposed_action: "a" } }; - expect(toResultRecord(r, "run-1/i1", "T")).toMatchObject({ - status: "paused", gate: { gateId: 1, summary: "s", proposed_action: "a" }, verdict: null, patch: null, + it('maps paused → gate-bearing record', () => { + const r: LeafResult = { + status: 'paused', + gateId: 1, + gate: { summary: 's', proposed_action: 'a' }, + }; + expect(toResultRecord(r, 'run-1/i1', 'T')).toMatchObject({ + status: 'paused', + gate: { gateId: 1, summary: 's', proposed_action: 'a' }, + verdict: null, + patch: null, }); }); - it("maps failed → reason-bearing record", () => { - const r: LeafResult = { status: "failed", reason: "no_verdict" }; - expect(toResultRecord(r, "s", "T")).toMatchObject({ status: "failed", reason: "no_verdict", patch: null }); + it('maps failed → reason-bearing record', () => { + const r: LeafResult = { status: 'failed', reason: 'no_verdict' }; + expect(toResultRecord(r, 's', 'T')).toMatchObject({ + status: 'failed', + reason: 'no_verdict', + patch: null, + }); }); }); -describe("toResultRecord — solved", () => { - it("carries the patch and sets status solved", () => { - const rec = toResultRecord({ status: "solved", patch: "diff --git a/x b/x\n" }, "run-1", "2026-07-14T00:00:00Z"); - expect(rec.status).toBe("solved"); - expect(rec.patch).toBe("diff --git a/x b/x\n"); +describe('toResultRecord — solved', () => { + it('carries the patch and sets status solved', () => { + const rec = toResultRecord( + { status: 'solved', patch: 'diff --git a/x b/x\n' }, + 'run-1', + '2026-07-14T00:00:00Z', + ); + expect(rec.status).toBe('solved'); + expect(rec.patch).toBe('diff --git a/x b/x\n'); expect(rec.verdict).toBeNull(); - expect(rec.sessionId).toBe("run-1"); + expect(rec.sessionId).toBe('run-1'); }); - it("carries token usage when the solved result has it", () => { + it('carries token usage when the solved result has it', () => { const rec = toResultRecord( - { status: "solved", patch: "d", usage: { input: 10, output: 5, cacheRead: 2, cacheWrite: 1, total: 18 } }, - "run-1", "t", + { + status: 'solved', + patch: 'd', + usage: { input: 10, output: 5, cacheRead: 2, cacheWrite: 1, total: 18 }, + }, + 'run-1', + 't', ); expect(rec.usage).toEqual({ input: 10, output: 5, cacheRead: 2, cacheWrite: 1, total: 18 }); }); - it("defaults usage to null when the solved result has none", () => { - expect(toResultRecord({ status: "solved", patch: "d" }, "run-1", "t").usage).toBeNull(); + it('defaults usage to null when the solved result has none', () => { + expect(toResultRecord({ status: 'solved', patch: 'd' }, 'run-1', 't').usage).toBeNull(); }); - it("defaults patch to null for non-solve results", () => { - const rec = toResultRecord({ status: "aborted" }, "run-1", "t"); + it('defaults patch to null for non-solve results', () => { + const rec = toResultRecord({ status: 'aborted' }, 'run-1', 't'); expect(rec.patch).toBeNull(); }); }); -describe("toResultRecord — responded", () => { - it("carries the text and sets status responded, leaving other payloads null", () => { - const rec = toResultRecord({ status: "responded", text: "hello world" }, "run-1/i1", "T"); - expect(rec.status).toBe("responded"); - expect(rec.text).toBe("hello world"); +describe('toResultRecord — responded', () => { + it('carries the text and sets status responded, leaving other payloads null', () => { + const rec = toResultRecord({ status: 'responded', text: 'hello world' }, 'run-1/i1', 'T'); + expect(rec.status).toBe('responded'); + expect(rec.text).toBe('hello world'); expect(rec.verdict).toBeNull(); expect(rec.gate).toBeNull(); expect(rec.reason).toBeNull(); expect(rec.patch).toBeNull(); expect(rec.usage).toBeNull(); - expect(rec.sessionId).toBe("run-1/i1"); + expect(rec.sessionId).toBe('run-1/i1'); }); - it("carries token usage when the responded result has it", () => { + it('carries token usage when the responded result has it', () => { const rec = toResultRecord( - { status: "responded", text: "t", usage: { input: 10, output: 5, cacheRead: 2, cacheWrite: 1, total: 18 } }, - "run-1", "t", + { + status: 'responded', + text: 't', + usage: { input: 10, output: 5, cacheRead: 2, cacheWrite: 1, total: 18 }, + }, + 'run-1', + 't', ); expect(rec.usage).toEqual({ input: 10, output: 5, cacheRead: 2, cacheWrite: 1, total: 18 }); }); - it("defaults text to null for non-prompt results", () => { - expect(toResultRecord({ status: "aborted" }, "run-1", "t").text).toBeNull(); + it('defaults text to null for non-prompt results', () => { + expect(toResultRecord({ status: 'aborted' }, 'run-1', 't').text).toBeNull(); }); }); -describe("writeResult / readResult", () => { - it("round-trips a record and sets the TTL", async () => { +describe('writeResult / readResult', () => { + it('round-trips a record and sets the TTL', async () => { const redis = fakeRedis(); - const rec: LeafResultRecord = { status: "done", verdict: { item_id: "i1", verdict: "CLEAR", reason: "ok" }, gate: null, reason: null, patch: null, text: null, sessionId: "run-1/i1", ts: "T" }; - await writeResult(redis, "run-1-i1", rec, 3600); - expect(redis.ttl.get("leaf:result:run-1-i1")).toBe(3600); - expect(await readResult(redis, "run-1-i1")).toEqual(rec); + const rec: LeafResultRecord = { + status: 'done', + verdict: { item_id: 'i1', verdict: 'CLEAR', reason: 'ok' }, + gate: null, + reason: null, + patch: null, + text: null, + sessionId: 'run-1/i1', + ts: 'T', + }; + await writeResult(redis, 'run-1-i1', rec, 3600); + expect(redis.ttl.get('leaf:result:run-1-i1')).toBe(3600); + expect(await readResult(redis, 'run-1-i1')).toEqual(rec); }); - it("returns null for a missing key", async () => { - expect(await readResult(fakeRedis(), "nope")).toBeNull(); + it('returns null for a missing key', async () => { + expect(await readResult(fakeRedis(), 'nope')).toBeNull(); }); - it("returns null for a garbled value", async () => { + it('returns null for a garbled value', async () => { const redis = fakeRedis(); - await redis.set("leaf:result:x", "{not json", {}); - expect(await readResult(redis, "x")).toBeNull(); + await redis.set('leaf:result:x', '{not json', {}); + expect(await readResult(redis, 'x')).toBeNull(); }); }); diff --git a/harness/test/model-gateway.test.ts b/harness/test/model-gateway.test.ts index 9f1b4be..e512373 100644 --- a/harness/test/model-gateway.test.ts +++ b/harness/test/model-gateway.test.ts @@ -1,9 +1,9 @@ -import { describe, it, expect, beforeEach, afterEach } from "vitest"; -import { applyModelGateway } from "../src/run-turn"; +import { describe, it, expect, beforeEach, afterEach } from 'vitest'; +import { applyModelGateway } from '../src/run-turn'; -const baseModel = { id: "claude-haiku-4-5", headers: { "x-api-key": "orig" } } as never; +const baseModel = { id: 'claude-haiku-4-5', headers: { 'x-api-key': 'orig' } } as never; -describe("applyModelGateway", () => { +describe('applyModelGateway', () => { let savedKey: string | undefined; let savedBase: string | undefined; let savedTok: string | undefined; @@ -16,67 +16,70 @@ describe("applyModelGateway", () => { delete process.env.ANTHROPIC_AUTH_TOKEN; }); afterEach(() => { - if (savedKey === undefined) delete process.env.ANTHROPIC_API_KEY; else process.env.ANTHROPIC_API_KEY = savedKey; - if (savedBase === undefined) delete process.env.ANTHROPIC_BASE_URL; else process.env.ANTHROPIC_BASE_URL = savedBase; - if (savedTok === undefined) delete process.env.ANTHROPIC_AUTH_TOKEN; else process.env.ANTHROPIC_AUTH_TOKEN = savedTok; + if (savedKey === undefined) delete process.env.ANTHROPIC_API_KEY; + else process.env.ANTHROPIC_API_KEY = savedKey; + if (savedBase === undefined) delete process.env.ANTHROPIC_BASE_URL; + else process.env.ANTHROPIC_BASE_URL = savedBase; + if (savedTok === undefined) delete process.env.ANTHROPIC_AUTH_TOKEN; + else process.env.ANTHROPIC_AUTH_TOKEN = savedTok; }); - it("returns the base model unchanged when no gateway base or token is set", () => { + it('returns the base model unchanged when no gateway base or token is set', () => { const m = applyModelGateway(baseModel, {}) as any; expect(m).toBe(baseModel); }); - it("applies the gateway baseUrl and Bearer auth, stripping x-api-key", () => { + it('applies the gateway baseUrl and Bearer auth, stripping x-api-key', () => { const m = applyModelGateway(baseModel, { - anthropicBaseUrl: "https://gw.example/v1", - anthropicAuthToken: "tok-123", + anthropicBaseUrl: 'https://gw.example/v1', + anthropicAuthToken: 'tok-123', }) as any; - expect(m.baseUrl).toBe("https://gw.example/v1"); - expect(m.headers.Authorization).toBe("Bearer tok-123"); - expect(m.headers["x-api-key"]).toBeNull(); + expect(m.baseUrl).toBe('https://gw.example/v1'); + expect(m.headers.Authorization).toBe('Bearer tok-123'); + expect(m.headers['x-api-key']).toBeNull(); }); - it("reads ANTHROPIC_BASE_URL / ANTHROPIC_AUTH_TOKEN from env when config omits them", () => { - process.env.ANTHROPIC_BASE_URL = "https://env-gw/v1"; - process.env.ANTHROPIC_AUTH_TOKEN = "env-tok"; + it('reads ANTHROPIC_BASE_URL / ANTHROPIC_AUTH_TOKEN from env when config omits them', () => { + process.env.ANTHROPIC_BASE_URL = 'https://env-gw/v1'; + process.env.ANTHROPIC_AUTH_TOKEN = 'env-tok'; const m = applyModelGateway(baseModel, {}) as any; - expect(m.baseUrl).toBe("https://env-gw/v1"); - expect(m.headers.Authorization).toBe("Bearer env-tok"); + expect(m.baseUrl).toBe('https://env-gw/v1'); + expect(m.headers.Authorization).toBe('Bearer env-tok'); }); - it("seeds ANTHROPIC_API_KEY from the auth token when the key is unset", () => { - applyModelGateway(baseModel, { anthropicAuthToken: "tok-xyz" }); - expect(process.env.ANTHROPIC_API_KEY).toBe("tok-xyz"); + it('seeds ANTHROPIC_API_KEY from the auth token when the key is unset', () => { + applyModelGateway(baseModel, { anthropicAuthToken: 'tok-xyz' }); + expect(process.env.ANTHROPIC_API_KEY).toBe('tok-xyz'); }); - it("disables gateway-incompatible compat flags when a gateway base is set", () => { + it('disables gateway-incompatible compat flags when a gateway base is set', () => { // litellm rejects per-tool eager_input_streaming / cache_control; the gateway model must // disable these so convertTools() omits them (otherwise tool-bearing requests 400). - const m = applyModelGateway(baseModel, { anthropicBaseUrl: "https://gw.example/v1" }) as any; + const m = applyModelGateway(baseModel, { anthropicBaseUrl: 'https://gw.example/v1' }) as any; expect(m.compat.supportsEagerToolInputStreaming).toBe(false); expect(m.compat.supportsCacheControlOnTools).toBe(false); expect(m.compat.supportsLongCacheRetention).toBe(false); }); - it("does not add compat when no gateway base is set (direct API)", () => { - const m = applyModelGateway(baseModel, { anthropicAuthToken: "tok-only" }) as any; + it('does not add compat when no gateway base is set (direct API)', () => { + const m = applyModelGateway(baseModel, { anthropicAuthToken: 'tok-only' }) as any; expect(m.compat).toBeUndefined(); }); - it("applies baseUrl + disables compat but sets no auth header for a token-less public gateway", () => { - const m = applyModelGateway(baseModel, { anthropicBaseUrl: "https://public-gw/v1" }) as any; - expect(m.baseUrl).toBe("https://public-gw/v1"); + it('applies baseUrl + disables compat but sets no auth header for a token-less public gateway', () => { + const m = applyModelGateway(baseModel, { anthropicBaseUrl: 'https://public-gw/v1' }) as any; + expect(m.baseUrl).toBe('https://public-gw/v1'); expect(m.compat.supportsEagerToolInputStreaming).toBe(false); // no token → no Authorization header, and the original headers are left untouched expect(m.headers.Authorization).toBeUndefined(); - expect(m.headers["x-api-key"]).toBe("orig"); + expect(m.headers['x-api-key']).toBe('orig'); }); - it("treats an empty-string config value as unset and falls back to the env var", () => { - process.env.ANTHROPIC_BASE_URL = "https://env-gw/v1"; - process.env.ANTHROPIC_AUTH_TOKEN = "env-tok"; - const m = applyModelGateway(baseModel, { anthropicBaseUrl: "", anthropicAuthToken: "" }) as any; - expect(m.baseUrl).toBe("https://env-gw/v1"); - expect(m.headers.Authorization).toBe("Bearer env-tok"); + it('treats an empty-string config value as unset and falls back to the env var', () => { + process.env.ANTHROPIC_BASE_URL = 'https://env-gw/v1'; + process.env.ANTHROPIC_AUTH_TOKEN = 'env-tok'; + const m = applyModelGateway(baseModel, { anthropicBaseUrl: '', anthropicAuthToken: '' }) as any; + expect(m.baseUrl).toBe('https://env-gw/v1'); + expect(m.headers.Authorization).toBe('Bearer env-tok'); }); }); diff --git a/harness/test/pool-live-smoke.test.ts b/harness/test/pool-live-smoke.test.ts index c25026a..8777218 100644 --- a/harness/test/pool-live-smoke.test.ts +++ b/harness/test/pool-live-smoke.test.ts @@ -1,13 +1,13 @@ // harness/test/pool-live-smoke.test.ts -import { describe, it, expect } from "vitest"; -import { RedisLeaseStore, leaseKey } from "../src/sandbox-lease.js"; +import { describe, it, expect } from 'vitest'; +import { RedisLeaseStore, leaseKey } from '../src/sandbox-lease.js'; // Gate: only runs with a live Redis (docker) and POOL_LIVE_SMOKE=1. -const live = process.env.POOL_LIVE_SMOKE === "1"; -const url = process.env.REDIS_URL ?? "redis://127.0.0.1:6379"; +const live = process.env.POOL_LIVE_SMOKE === '1'; +const url = process.env.REDIS_URL ?? 'redis://127.0.0.1:6379'; -describe.skipIf(!live)("RedisLeaseStore (live redis)", () => { - it("enforces the soft cap and reclaims expired leases", async () => { +describe.skipIf(!live)('RedisLeaseStore (live redis)', () => { + it('enforces the soft cap and reclaims expired leases', async () => { const pod = `smoke-${process.pid}`; // Controllable clock so we can expire a lease deterministically. let now = 1_000_000; @@ -15,18 +15,18 @@ describe.skipIf(!live)("RedisLeaseStore (live redis)", () => { try { // Fresh key. // @ts-expect-error reach the raw client for test cleanup only - await store["client"].del(leaseKey(pod)); + await store['client'].del(leaseKey(pod)); - expect(await store.acquire(pod, 2, "a", 1000)).toBe(true); - expect(await store.acquire(pod, 2, "b", 1000)).toBe(true); - expect(await store.acquire(pod, 2, "c", 1000)).toBe(false); // at cap + expect(await store.acquire(pod, 2, 'a', 1000)).toBe(true); + expect(await store.acquire(pod, 2, 'b', 1000)).toBe(true); + expect(await store.acquire(pod, 2, 'c', 1000)).toBe(false); // at cap expect(await store.load(pod)).toBe(2); now += 2000; // both leases expire - expect(await store.acquire(pod, 2, "c", 1000)).toBe(true); // reclaimed slot + expect(await store.acquire(pod, 2, 'c', 1000)).toBe(true); // reclaimed slot expect(await store.load(pod)).toBe(1); } finally { - await store.release(pod, "c"); + await store.release(pod, 'c'); await store.close(); } }); diff --git a/harness/test/pool-records.test.ts b/harness/test/pool-records.test.ts index 621932d..242fc44 100644 --- a/harness/test/pool-records.test.ts +++ b/harness/test/pool-records.test.ts @@ -1,27 +1,27 @@ -import { afterEach, describe, expect, it } from "vitest"; -import { RedisRecordStore, type SandboxRecord } from "../src/pool-records.js"; +import { afterEach, describe, expect, it } from 'vitest'; +import { RedisRecordStore, type SandboxRecord } from '../src/pool-records.js'; const rec: SandboxRecord = { - sandboxId: "sbx-remote-1", - labels: { team: "t1" }, - capabilities: ["python3"], + sandboxId: 'sbx-remote-1', + labels: { team: 't1' }, + capabilities: ['python3'], capacityMax: 4, - transport: "grpc", + transport: 'grpc', }; -describe("RedisRecordStore", () => { +describe('RedisRecordStore', () => { const store = new RedisRecordStore(); afterEach(async () => { await store.remove(rec.sandboxId); }); - it("put then list returns the record", async () => { + it('put then list returns the record', async () => { await store.put(rec); const all = await store.list(); expect(all.find((r) => r.sandboxId === rec.sandboxId)).toEqual(rec); }); - it("remove drops it from list", async () => { + it('remove drops it from list', async () => { await store.put(rec); await store.remove(rec.sandboxId); const all = await store.list(); diff --git a/harness/test/request-approval-tool.test.ts b/harness/test/request-approval-tool.test.ts index b1f3497..62568b0 100644 --- a/harness/test/request-approval-tool.test.ts +++ b/harness/test/request-approval-tool.test.ts @@ -1,51 +1,86 @@ // harness/test/request-approval-tool.test.ts -import { describe, it, expect } from "vitest"; -import { requestApprovalExtension, type GateCapture } from "../src/request-approval-tool"; +import { describe, it, expect } from 'vitest'; +import { requestApprovalExtension, type GateCapture } from '../src/request-approval-tool'; function fakePi() { const tools: any[] = []; return { api: { registerTool: (t: any) => tools.push(t), on: () => {} } as any, tools }; } -describe("requestApprovalExtension", () => { - it("registers a request_approval tool", () => { +describe('requestApprovalExtension', () => { + it('registers a request_approval tool', () => { const { api, tools } = fakePi(); requestApprovalExtension({}, undefined, 0)(api); expect(tools).toHaveLength(1); - expect(tools[0].name).toBe("request_approval"); + expect(tools[0].name).toBe('request_approval'); }); - it("captures the gate (with nextGateId) and appends a durable gate-request entry", async () => { + it('captures the gate (with nextGateId) and appends a durable gate-request entry', async () => { const capture: GateCapture = {}; const appended: Array<{ t: string; d: unknown }> = []; - const sink = { appendCustomEntry: (t: string, d?: unknown) => { appended.push({ t, d }); return "id"; } }; + const sink = { + appendCustomEntry: (t: string, d?: unknown) => { + appended.push({ t, d }); + return 'id'; + }, + }; const { api, tools } = fakePi(); requestApprovalExtension(capture, sink, 3)(api); - const res = await tools[0].execute("call-1", { summary: "did X", proposed_action: "do Y" }, undefined, undefined, {} as any); - expect(capture.gate).toEqual({ gateId: 3, summary: "did X", proposed_action: "do Y" }); - expect(appended).toEqual([{ t: "gate-request", d: { gateId: 3, summary: "did X", proposed_action: "do Y" } }]); + const res = await tools[0].execute( + 'call-1', + { summary: 'did X', proposed_action: 'do Y' }, + undefined, + undefined, + {} as any, + ); + expect(capture.gate).toEqual({ gateId: 3, summary: 'did X', proposed_action: 'do Y' }); + expect(appended).toEqual([ + { t: 'gate-request', d: { gateId: 3, summary: 'did X', proposed_action: 'do Y' } }, + ]); expect(res.isError).toBeFalsy(); }); - it("rejects empty summary/proposed_action and does not capture or append", async () => { + it('rejects empty summary/proposed_action and does not capture or append', async () => { const capture: GateCapture = {}; const appended: unknown[] = []; - const sink = { appendCustomEntry: (t: string, d?: unknown) => { appended.push({ t, d }); return "id"; } }; + const sink = { + appendCustomEntry: (t: string, d?: unknown) => { + appended.push({ t, d }); + return 'id'; + }, + }; const { api, tools } = fakePi(); requestApprovalExtension(capture, sink, 0)(api); - const res = await tools[0].execute("c", { summary: "", proposed_action: "y" }, undefined, undefined, {} as any); + const res = await tools[0].execute( + 'c', + { summary: '', proposed_action: 'y' }, + undefined, + undefined, + {} as any, + ); expect(capture.gate).toBeUndefined(); expect(appended).toHaveLength(0); expect(res.isError).toBe(true); }); - it("rejects an empty proposed_action (valid summary) and does not capture or append", async () => { + it('rejects an empty proposed_action (valid summary) and does not capture or append', async () => { const capture: GateCapture = {}; const appended: unknown[] = []; - const sink = { appendCustomEntry: (t: string, d?: unknown) => { appended.push({ t, d }); return "id"; } }; + const sink = { + appendCustomEntry: (t: string, d?: unknown) => { + appended.push({ t, d }); + return 'id'; + }, + }; const { api, tools } = fakePi(); requestApprovalExtension(capture, sink, 0)(api); - const res = await tools[0].execute("c", { summary: "did X", proposed_action: "" }, undefined, undefined, {} as any); + const res = await tools[0].execute( + 'c', + { summary: 'did X', proposed_action: '' }, + undefined, + undefined, + {} as any, + ); expect(capture.gate).toBeUndefined(); expect(appended).toHaveLength(0); expect(res.isError).toBe(true); diff --git a/harness/test/run-leaf.test.ts b/harness/test/run-leaf.test.ts index aab06cc..639ba39 100644 --- a/harness/test/run-leaf.test.ts +++ b/harness/test/run-leaf.test.ts @@ -1,4 +1,4 @@ -import { describe, it, expect, vi } from "vitest"; +import { describe, it, expect, vi } from 'vitest'; // realProduceVerdict (exercised via the exported runLeaf() below, with no `deps.produceVerdict` // override) drives real Redis/Pi/model machinery in production. Mock those module boundaries so @@ -12,12 +12,12 @@ const { selectPoolSandboxMock, FakeSandboxPoolSaturatedError } = vi.hoisted(() = class FakeSandboxPoolSaturatedError extends Error { constructor(selector: string) { super(`sandbox pool '${selector}' saturated: all pods at capacity`); - this.name = "SandboxPoolSaturatedError"; + this.name = 'SandboxPoolSaturatedError'; } } return { selectPoolSandboxMock: vi.fn(), FakeSandboxPoolSaturatedError }; }); -vi.mock("../src/select-sandbox.js", () => ({ +vi.mock('../src/select-sandbox.js', () => ({ selectPoolSandbox: (...args: unknown[]) => selectPoolSandboxMock(...args), SandboxPoolSaturatedError: FakeSandboxPoolSaturatedError, })); @@ -25,34 +25,46 @@ vi.mock("../src/select-sandbox.js", () => ({ const { k8sSandboxExtensionMock, kubectlTransportMock } = vi.hoisted(() => ({ k8sSandboxExtensionMock: vi.fn(() => () => {}), kubectlTransportMock: vi.fn(() => ({ - exec: vi.fn(async () => ({ stdout: Buffer.from(""), exitCode: 0, truncated: false })), + exec: vi.fn(async () => ({ stdout: Buffer.from(''), exitCode: 0, truncated: false })), close: vi.fn(async () => {}), })), })); -vi.mock("@sh/k8s-sandbox", () => ({ +vi.mock('@sh/k8s-sandbox', () => ({ k8sSandboxExtension: (...args: unknown[]) => k8sSandboxExtensionMock(...args), KubectlTransport: (...args: unknown[]) => kubectlTransportMock(...args), })); const { FakeRedisSessionBackend } = vi.hoisted(() => { class FakeRedisSessionBackend { - async read(_sid: string) { return []; } - async latestWhere(_sid: string, _pred: unknown) { return null; } - async append(_sid: string, _entry: unknown, _piType: string) { return {}; } - async list() { return []; } + async read(_sid: string) { + return []; + } + async latestWhere(_sid: string, _pred: unknown) { + return null; + } + async append(_sid: string, _entry: unknown, _piType: string) { + return {}; + } + async list() { + return []; + } async close() {} } return { FakeRedisSessionBackend }; }); -vi.mock("@sh/session-backend", () => ({ +vi.mock('@sh/session-backend', () => ({ RedisSessionBackend: FakeRedisSessionBackend, })); const { FakeSessionManager, FakeResourceLoader, createAgentSessionMock } = vi.hoisted(() => { class FakeSessionManager { constructor(private sid: string) {} - getSessionId() { return this.sid; } - appendCustomEntry(_type: string, _data?: unknown) { return "entry-id"; } + getSessionId() { + return this.sid; + } + appendCustomEntry(_type: string, _data?: unknown) { + return 'entry-id'; + } } class FakeResourceLoader { constructor(public opts: unknown) {} @@ -64,63 +76,75 @@ const { FakeSessionManager, FakeResourceLoader, createAgentSessionMock } = vi.ho createAgentSessionMock: vi.fn(async () => ({ session: { prompt: async () => {} } })), }; }); -vi.mock("@earendil-works/pi-coding-agent", () => ({ +vi.mock('@earendil-works/pi-coding-agent', () => ({ createAgentSession: (...args: unknown[]) => createAgentSessionMock(...args), DefaultResourceLoader: FakeResourceLoader, - getAgentDir: () => "/fake/agent-dir", + getAgentDir: () => '/fake/agent-dir', SessionManager: { - create: (_cwd: string, _snapshot: unknown, opts: { id: string }) => new FakeSessionManager(opts.id), + create: (_cwd: string, _snapshot: unknown, opts: { id: string }) => + new FakeSessionManager(opts.id), openFromCheckpoint: async (sid: string) => new FakeSessionManager(sid), }, SettingsManager: { create: () => ({}) }, })); -import { runLeaf, buildLeafPrompt, buildSolvePrompt, leafSessionId, validateItem } from "../src/run-leaf.js"; -import type { LeafEnvelope } from "../src/run-leaf.js"; -import { SandboxPoolSaturatedError } from "../src/select-sandbox.js"; - -describe("LeafEnvelope repo ref fields", () => { - it("accepts optional repoUrl and ref", () => { +import { + runLeaf, + buildLeafPrompt, + buildSolvePrompt, + leafSessionId, + validateItem, +} from '../src/run-leaf.js'; +import type { LeafEnvelope } from '../src/run-leaf.js'; +import { SandboxPoolSaturatedError } from '../src/select-sandbox.js'; + +describe('LeafEnvelope repo ref fields', () => { + it('accepts optional repoUrl and ref', () => { const env: LeafEnvelope = { - sessionId: "run-a/item-1", - item: { item_id: "item-1", file: "a.ts", pattern: "x" }, - repoUrl: "https://git.example/r.git", - ref: "abc123", + sessionId: 'run-a/item-1', + item: { item_id: 'item-1', file: 'a.ts', pattern: 'x' }, + repoUrl: 'https://git.example/r.git', + ref: 'abc123', }; - expect(env.repoUrl).toBe("https://git.example/r.git"); - expect(env.ref).toBe("abc123"); + expect(env.repoUrl).toBe('https://git.example/r.git'); + expect(env.ref).toBe('abc123'); }); }); -describe("LeafEnvelope prompt fields", () => { - it("accepts kind:prompt with a prompt string", () => { +describe('LeafEnvelope prompt fields', () => { + it('accepts kind:prompt with a prompt string', () => { const env: LeafEnvelope = { - sessionId: "run-a/item-1", - item: { item_id: "item-1", file: "a.ts", pattern: "x" }, - kind: "prompt", - prompt: "Summarize the repo.", + sessionId: 'run-a/item-1', + item: { item_id: 'item-1', file: 'a.ts', pattern: 'x' }, + kind: 'prompt', + prompt: 'Summarize the repo.', }; - expect(env.kind).toBe("prompt"); - expect(env.prompt).toBe("Summarize the repo."); + expect(env.kind).toBe('prompt'); + expect(env.prompt).toBe('Summarize the repo.'); }); }); -describe("validateItem", () => { - it("accepts a well-formed item", () => { - expect(validateItem({ item_id: "i", file: "f", pattern: "p" })).toEqual({ item_id: "i", file: "f", pattern: "p", require_approval: false }); +describe('validateItem', () => { + it('accepts a well-formed item', () => { + expect(validateItem({ item_id: 'i', file: 'f', pattern: 'p' })).toEqual({ + item_id: 'i', + file: 'f', + pattern: 'p', + require_approval: false, + }); }); - it("rejects a missing field and non-objects", () => { - expect(validateItem({ item_id: "i", file: "f" })).toBeNull(); + it('rejects a missing field and non-objects', () => { + expect(validateItem({ item_id: 'i', file: 'f' })).toBeNull(); expect(validateItem(null)).toBeNull(); }); }); -describe("buildLeafPrompt", () => { - it("includes the file, pattern, and submit_verdict instruction", () => { - const p = buildLeafPrompt({ item_id: "i1", file: "a.py", pattern: "eval(" }); - expect(p).toContain("a.py"); - expect(p).toContain("eval("); - expect(p).toContain("submit_verdict"); +describe('buildLeafPrompt', () => { + it('includes the file, pattern, and submit_verdict instruction', () => { + const p = buildLeafPrompt({ item_id: 'i1', file: 'a.py', pattern: 'eval(' }); + expect(p).toContain('a.py'); + expect(p).toContain('eval('); + expect(p).toContain('submit_verdict'); }); }); @@ -129,8 +153,8 @@ describe("buildLeafPrompt", () => { // LeafEnvelope -- i.e. straight off the request body, with no normalisation in between -- so the // input is caller-controlled. These cases pin the rewrite to the old regex's exact behaviour; the // last is the regression guard: the regex burns ~15s of CPU on 100k slashes, the scan ~0.005ms. -describe("prompt builders: trailing-slash strip on workspaceRef", () => { - const item = { item_id: "i1", file: "a.py", pattern: "eval(" }; +describe('prompt builders: trailing-slash strip on workspaceRef', () => { + const item = { item_id: 'i1', file: 'a.py', pattern: 'eval(' }; // prettier-ignore const refs = [ "/w", "/w/", "/w//", "/w/////////", "/", "////", "/w/x", "/w ", "/w /", "/wörk/", "/w/./", @@ -138,23 +162,23 @@ describe("prompt builders: trailing-slash strip on workspaceRef", () => { for (const ref of refs) { it(`buildLeafPrompt matches the old strip for ${JSON.stringify(ref)}`, () => { - expect(buildLeafPrompt(item, ref)).toContain(`${ref.replace(/\/+$/, "")}/a.py`); + expect(buildLeafPrompt(item, ref)).toContain(`${ref.replace(/\/+$/, '')}/a.py`); }); it(`buildSolvePrompt matches the old strip for ${JSON.stringify(ref)}`, () => { - expect(buildSolvePrompt("stmt", ref)).toContain( - `root (an absolute path in your sandbox): ${ref.replace(/\/+$/, "")}`, + expect(buildSolvePrompt('stmt', ref)).toContain( + `root (an absolute path in your sandbox): ${ref.replace(/\/+$/, '')}`, ); }); } - it("leaves the bare file name alone when workspaceRef is empty", () => { - expect(buildLeafPrompt(item, "")).toContain("read tool): a.py"); + it('leaves the bare file name alone when workspaceRef is empty', () => { + expect(buildLeafPrompt(item, '')).toContain('read tool): a.py'); }); - it("handles a pathological run of slashes in linear time (js/polynomial-redos guard)", () => { + it('handles a pathological run of slashes in linear time (js/polynomial-redos guard)', () => { // Many slashes then a non-slash: nothing to strip, but `/\/+$/` retries from every position. - const evil = `/w${"/".repeat(100_000)}x`; + const evil = `/w${'/'.repeat(100_000)}x`; const started = performance.now(); const prompt = buildLeafPrompt(item, evil); expect(performance.now() - started).toBeLessThan(1_000); @@ -162,91 +186,125 @@ describe("prompt builders: trailing-slash strip on workspaceRef", () => { }); }); -describe("buildSolvePrompt", () => { - it("embeds the problem statement and the absolute worktree root", () => { - const p = buildSolvePrompt("Fix the off-by-one in paginate().", "/workspace/leaves/run-1/"); - expect(p).toContain("Fix the off-by-one in paginate()."); +describe('buildSolvePrompt', () => { + it('embeds the problem statement and the absolute worktree root', () => { + const p = buildSolvePrompt('Fix the off-by-one in paginate().', '/workspace/leaves/run-1/'); + expect(p).toContain('Fix the off-by-one in paginate().'); // trailing slash trimmed; root given as an absolute path - expect(p).toContain("/workspace/leaves/run-1"); - expect(p).not.toContain("/workspace/leaves/run-1/\n"); + expect(p).toContain('/workspace/leaves/run-1'); + expect(p).not.toContain('/workspace/leaves/run-1/\n'); // solve prompt must NOT instruct submit_verdict (that is the converge path) - expect(p).not.toContain("submit_verdict"); + expect(p).not.toContain('submit_verdict'); }); }); -describe("runLeaf", () => { - it("fails with bad_inputs when item is missing", async () => { - const r = await runLeaf({ sessionId: "s" } as any, undefined, { produceVerdict: async () => {} }); - expect(r).toEqual({ status: "failed", reason: "bad_inputs" }); +describe('runLeaf', () => { + it('fails with bad_inputs when item is missing', async () => { + const r = await runLeaf({ sessionId: 's' } as any, undefined, { + produceVerdict: async () => {}, + }); + expect(r).toEqual({ status: 'failed', reason: 'bad_inputs' }); }); - it("returns the verdict inline on success", async () => { - const env = { sessionId: "run/i1", item: { item_id: "i1", file: "f", pattern: "p" } }; + it('returns the verdict inline on success', async () => { + const env = { sessionId: 'run/i1', item: { item_id: 'i1', file: 'f', pattern: 'p' } }; const r = await runLeaf(env, undefined, { - produceVerdict: async (_i, _e, _c, cap) => { cap.verdict = { item_id: "i1", verdict: "FLAGGED", reason: "x" }; }, + produceVerdict: async (_i, _e, _c, cap) => { + cap.verdict = { item_id: 'i1', verdict: 'FLAGGED', reason: 'x' }; + }, + }); + expect(r).toEqual({ + status: 'done', + verdict: { item_id: 'i1', verdict: 'FLAGGED', reason: 'x' }, }); - expect(r).toEqual({ status: "done", verdict: { item_id: "i1", verdict: "FLAGGED", reason: "x" } }); }); - it("returns the gate inline when paused", async () => { - const env = { sessionId: "run/i1", item: { item_id: "i1", file: "f", pattern: "p", require_approval: true } }; + it('returns the gate inline when paused', async () => { + const env = { + sessionId: 'run/i1', + item: { item_id: 'i1', file: 'f', pattern: 'p', require_approval: true }, + }; const r = await runLeaf(env, undefined, { - produceVerdict: async (_i, _e, _c, cap) => { cap.gate = { gateId: 2, summary: "s", proposed_action: "a" }; }, + produceVerdict: async (_i, _e, _c, cap) => { + cap.gate = { gateId: 2, summary: 's', proposed_action: 'a' }; + }, + }); + expect(r).toEqual({ + status: 'paused', + gateId: 2, + gate: { summary: 's', proposed_action: 'a' }, }); - expect(r).toEqual({ status: "paused", gateId: 2, gate: { summary: "s", proposed_action: "a" } }); }); - it("returns aborted when the capture is aborted", async () => { - const env = { sessionId: "run/i1", item: { item_id: "i1", file: "f", pattern: "p" } }; - const r = await runLeaf(env, undefined, { produceVerdict: async (_i, _e, _c, cap) => { cap.aborted = true; } }); - expect(r).toEqual({ status: "aborted" }); + it('returns aborted when the capture is aborted', async () => { + const env = { sessionId: 'run/i1', item: { item_id: 'i1', file: 'f', pattern: 'p' } }; + const r = await runLeaf(env, undefined, { + produceVerdict: async (_i, _e, _c, cap) => { + cap.aborted = true; + }, + }); + expect(r).toEqual({ status: 'aborted' }); }); - it("fails with no_verdict when nothing is captured", async () => { - const env = { sessionId: "run/i1", item: { item_id: "i1", file: "f", pattern: "p" } }; + it('fails with no_verdict when nothing is captured', async () => { + const env = { sessionId: 'run/i1', item: { item_id: 'i1', file: 'f', pattern: 'p' } }; const r = await runLeaf(env, undefined, { produceVerdict: async () => {} }); - expect(r).toEqual({ status: "failed", reason: "no_verdict" }); + expect(r).toEqual({ status: 'failed', reason: 'no_verdict' }); }); - it("fails with invalid_verdict when the captured verdict is off-shape", async () => { - const env = { sessionId: "run/i1", item: { item_id: "i1", file: "f", pattern: "p" } }; - const r = await runLeaf(env, undefined, { produceVerdict: async (_i, _e, _c, cap) => { cap.verdict = { item_id: "i1", verdict: "MAYBE", reason: "x" } as any; } }); - expect(r.status).toBe("failed"); - if (r.status === "failed") expect(r.reason).toBe("invalid_verdict"); + it('fails with invalid_verdict when the captured verdict is off-shape', async () => { + const env = { sessionId: 'run/i1', item: { item_id: 'i1', file: 'f', pattern: 'p' } }; + const r = await runLeaf(env, undefined, { + produceVerdict: async (_i, _e, _c, cap) => { + cap.verdict = { item_id: 'i1', verdict: 'MAYBE', reason: 'x' } as any; + }, + }); + expect(r.status).toBe('failed'); + if (r.status === 'failed') expect(r.reason).toBe('invalid_verdict'); }); - it("returns failed:error when produceVerdict throws", async () => { - const env = { sessionId: "run/i1", item: { item_id: "i1", file: "f", pattern: "p" } }; - const produceVerdict = async () => { throw new Error("boom"); }; + it('returns failed:error when produceVerdict throws', async () => { + const env = { sessionId: 'run/i1', item: { item_id: 'i1', file: 'f', pattern: 'p' } }; + const produceVerdict = async () => { + throw new Error('boom'); + }; const r = await runLeaf(env, undefined, { produceVerdict }); - expect(r.status).toBe("failed"); - if (r.status === "failed") expect(r.reason).toBe("error"); + expect(r.status).toBe('failed'); + if (r.status === 'failed') expect(r.reason).toBe('error'); }); - it("returns failed:saturated (not error) when the pool is saturated", async () => { + it('returns failed:saturated (not error) when the pool is saturated', async () => { // Distinguishing saturation from a generic error lets the sync /runs path implement the // spec §4.3 bounded-wait + 503 Retry-After behavior without touching the async path. - const env = { sessionId: "run/i1", item: { item_id: "i1", file: "f", pattern: "p" } }; - const produceVerdict = async () => { throw new SandboxPoolSaturatedError("pool=x"); }; + const env = { sessionId: 'run/i1', item: { item_id: 'i1', file: 'f', pattern: 'p' } }; + const produceVerdict = async () => { + throw new SandboxPoolSaturatedError('pool=x'); + }; const r = await runLeaf(env, undefined, { produceVerdict }); - expect(r.status).toBe("failed"); - if (r.status === "failed") expect(r.reason).toBe("saturated"); + expect(r.status).toBe('failed'); + if (r.status === 'failed') expect(r.reason).toBe('saturated'); }); }); -describe("leafSessionId", () => { - it("sanitizes the bare sessionId when no tenant is set", () => { - expect(leafSessionId({ sessionId: "run-1/i1" })).toBe("run-1-i1"); +describe('leafSessionId', () => { + it('sanitizes the bare sessionId when no tenant is set', () => { + expect(leafSessionId({ sessionId: 'run-1/i1' })).toBe('run-1-i1'); }); - it("prefixes and sanitizes with the tenant for per-tenant id isolation", () => { - expect(leafSessionId({ sessionId: "run-1/i1", tenant: "acme" })).toBe("acme-run-1-i1"); + it('prefixes and sanitizes with the tenant for per-tenant id isolation', () => { + expect(leafSessionId({ sessionId: 'run-1/i1', tenant: 'acme' })).toBe('acme-run-1-i1'); }); }); -describe("realProduceVerdict transport wiring (Task 9)", () => { - const FAKE_CONFIG = { pod: "sandbox-0", namespace: "default", context: undefined, podCwd: "/workspace", headCwd: "/head" }; +describe('realProduceVerdict transport wiring (Task 9)', () => { + const FAKE_CONFIG = { + pod: 'sandbox-0', + namespace: 'default', + context: undefined, + podCwd: '/workspace', + headCwd: '/head', + }; - it("pod path: builds a fresh KubectlTransport per phase and passes no transport to the extension", async () => { + it('pod path: builds a fresh KubectlTransport per phase and passes no transport to the extension', async () => { selectPoolSandboxMock.mockReset().mockResolvedValue({ config: FAKE_CONFIG, heartbeat: vi.fn(async () => {}), @@ -256,10 +314,10 @@ describe("realProduceVerdict transport wiring (Task 9)", () => { k8sSandboxExtensionMock.mockClear(); const env: LeafEnvelope = { - sessionId: "run/pod-1", - item: { item_id: "i1", file: "f", pattern: "p" }, - repoUrl: "https://git.example/r.git", - ref: "abc123", + sessionId: 'run/pod-1', + item: { item_id: 'i1', file: 'f', pattern: 'p' }, + repoUrl: 'https://git.example/r.git', + ref: 'abc123', }; await runLeaf(env); @@ -272,10 +330,13 @@ describe("realProduceVerdict transport wiring (Task 9)", () => { expect(result.value.close).toHaveBeenCalledTimes(1); } - expect(k8sSandboxExtensionMock).toHaveBeenCalledWith({ config: FAKE_CONFIG, transport: undefined }); + expect(k8sSandboxExtensionMock).toHaveBeenCalledWith({ + config: FAKE_CONFIG, + transport: undefined, + }); }); - it("uses a request-scoped sandbox pool selector", async () => { + it('uses a request-scoped sandbox pool selector', async () => { selectPoolSandboxMock.mockReset().mockResolvedValue({ config: FAKE_CONFIG, heartbeat: vi.fn(async () => {}), @@ -283,23 +344,25 @@ describe("realProduceVerdict transport wiring (Task 9)", () => { }); await runLeaf({ - sessionId: "run/workload-1", - sandboxPoolSelector: "sh.kagenti.io/sandbox-pool=workload-1", - item: { item_id: "i1", file: "f", pattern: "p" }, + sessionId: 'run/workload-1', + sandboxPoolSelector: 'sh.kagenti.io/sandbox-pool=workload-1', + item: { item_id: 'i1', file: 'f', pattern: 'p' }, }); expect(selectPoolSandboxMock).toHaveBeenCalledWith( - expect.objectContaining({ KAGENTI_SANDBOX_POOL_SELECTOR: "sh.kagenti.io/sandbox-pool=workload-1" }), + expect.objectContaining({ + KAGENTI_SANDBOX_POOL_SELECTOR: 'sh.kagenti.io/sandbox-pool=workload-1', + }), expect.any(String), expect.any(String), expect.any(Object), ); }); - it("grpc path: reuses selected.transport for converge + cleanup and closes it exactly once", async () => { + it('grpc path: reuses selected.transport for converge + cleanup and closes it exactly once', async () => { const close = vi.fn(async () => {}); const transport = { - exec: vi.fn(async () => ({ stdout: Buffer.from(""), exitCode: 0, truncated: false })), + exec: vi.fn(async () => ({ stdout: Buffer.from(''), exitCode: 0, truncated: false })), close, }; selectPoolSandboxMock.mockReset().mockResolvedValue({ @@ -312,10 +375,10 @@ describe("realProduceVerdict transport wiring (Task 9)", () => { k8sSandboxExtensionMock.mockClear(); const env: LeafEnvelope = { - sessionId: "run/grpc-1", - item: { item_id: "i1", file: "f", pattern: "p" }, - repoUrl: "https://git.example/r.git", - ref: "abc123", + sessionId: 'run/grpc-1', + item: { item_id: 'i1', file: 'f', pattern: 'p' }, + repoUrl: 'https://git.example/r.git', + ref: 'abc123', }; await runLeaf(env); @@ -327,169 +390,266 @@ describe("realProduceVerdict transport wiring (Task 9)", () => { }); }); -describe("buildLeafPrompt with require_approval", () => { - it("adds a request_approval instruction when the item requires approval", () => { - const p = buildLeafPrompt({ item_id: "i1", file: "a.py", pattern: "eval(", require_approval: true }); - expect(p).toContain("request_approval"); - }); - it("withholds the submit_verdict instruction in the gated turn (verdict comes after approval)", () => { - const p = buildLeafPrompt({ item_id: "i1", file: "a.py", pattern: "eval(", require_approval: true }); - expect(p).not.toContain("submit_verdict"); +describe('buildLeafPrompt with require_approval', () => { + it('adds a request_approval instruction when the item requires approval', () => { + const p = buildLeafPrompt({ + item_id: 'i1', + file: 'a.py', + pattern: 'eval(', + require_approval: true, + }); + expect(p).toContain('request_approval'); + }); + it('withholds the submit_verdict instruction in the gated turn (verdict comes after approval)', () => { + const p = buildLeafPrompt({ + item_id: 'i1', + file: 'a.py', + pattern: 'eval(', + require_approval: true, + }); + expect(p).not.toContain('submit_verdict'); }); - it("omits the gate instruction by default", () => { - const p = buildLeafPrompt({ item_id: "i1", file: "a.py", pattern: "eval(" }); - expect(p).not.toContain("request_approval"); + it('omits the gate instruction by default', () => { + const p = buildLeafPrompt({ item_id: 'i1', file: 'a.py', pattern: 'eval(' }); + expect(p).not.toContain('request_approval'); }); }); -describe("runLeaf — solve routing", () => { +describe('runLeaf — solve routing', () => { const base: LeafEnvelope = { - sessionId: "run-1", item: { item_id: "x", file: "f", pattern: "p" }, - kind: "solve", problemStatement: "do the thing", repoUrl: "git://x/repo.git", ref: "work", + sessionId: 'run-1', + item: { item_id: 'x', file: 'f', pattern: 'p' }, + kind: 'solve', + problemStatement: 'do the thing', + repoUrl: 'git://x/repo.git', + ref: 'work', }; - it("maps a captured patch to status solved", async () => { - const r = await runLeaf(base, undefined, { produceSolve: async (_e, _c, cap) => { cap.patch = "PATCH"; } }); - expect(r).toEqual({ status: "solved", patch: "PATCH" }); + it('maps a captured patch to status solved', async () => { + const r = await runLeaf(base, undefined, { + produceSolve: async (_e, _c, cap) => { + cap.patch = 'PATCH'; + }, + }); + expect(r).toEqual({ status: 'solved', patch: 'PATCH' }); }); - it("treats an unset patch as an empty (still solved) patch", async () => { - const r = await runLeaf(base, undefined, { produceSolve: async () => { /* no edits */ } }); - expect(r).toEqual({ status: "solved", patch: "" }); + it('treats an unset patch as an empty (still solved) patch', async () => { + const r = await runLeaf(base, undefined, { + produceSolve: async () => { + /* no edits */ + }, + }); + expect(r).toEqual({ status: 'solved', patch: '' }); }); - it("fails bad_inputs when problemStatement/repoUrl/ref are missing", async () => { - const r = await runLeaf({ sessionId: "s", item: base.item, kind: "solve" }); - expect(r).toEqual({ status: "failed", reason: "bad_inputs" }); + it('fails bad_inputs when problemStatement/repoUrl/ref are missing', async () => { + const r = await runLeaf({ sessionId: 's', item: base.item, kind: 'solve' }); + expect(r).toEqual({ status: 'failed', reason: 'bad_inputs' }); }); - it("maps pool saturation to a saturated failure", async () => { + it('maps pool saturation to a saturated failure', async () => { const r = await runLeaf(base, undefined, { - produceSolve: async () => { throw new SandboxPoolSaturatedError("full"); }, + produceSolve: async () => { + throw new SandboxPoolSaturatedError('full'); + }, }); - expect(r.status).toBe("failed"); - expect((r as { reason?: string }).reason).toBe("saturated"); + expect(r.status).toBe('failed'); + expect((r as { reason?: string }).reason).toBe('saturated'); }); }); -describe("runLeaf — prompt routing", () => { +describe('runLeaf — prompt routing', () => { const base: LeafEnvelope = { - sessionId: "run-1/i1", item: { item_id: "x", file: "f", pattern: "p" }, - kind: "prompt", prompt: "Summarize the repo.", + sessionId: 'run-1/i1', + item: { item_id: 'x', file: 'f', pattern: 'p' }, + kind: 'prompt', + prompt: 'Summarize the repo.', }; - it("maps end_turn → responded with the assistant text and usage", async () => { + it('maps end_turn → responded with the assistant text and usage', async () => { const executeTurn = vi.fn(async () => ({ - sessionId: "run-1-i1", response: "here is a summary", stopReason: "end_turn", + sessionId: 'run-1-i1', + response: 'here is a summary', + stopReason: 'end_turn', usage: { input: 3, output: 7, cacheRead: 0, cacheWrite: 0, total: 10 }, })); const r = await runLeaf(base, undefined, { executeTurn }); - expect(r).toEqual({ status: "responded", text: "here is a summary", usage: { input: 3, output: 7, cacheRead: 0, cacheWrite: 0, total: 10 } }); - expect(executeTurn).toHaveBeenCalledWith(expect.objectContaining({ - prompt: "Summarize the repo.", sessionId: "run-1/i1", createIfAbsent: true, - })); + expect(r).toEqual({ + status: 'responded', + text: 'here is a summary', + usage: { input: 3, output: 7, cacheRead: 0, cacheWrite: 0, total: 10 }, + }); + expect(executeTurn).toHaveBeenCalledWith( + expect.objectContaining({ + prompt: 'Summarize the repo.', + sessionId: 'run-1/i1', + createIfAbsent: true, + }), + ); }); - it("maps a non-terminal stopReason (max_tokens) → responded", async () => { + it('maps a non-terminal stopReason (max_tokens) → responded', async () => { const executeTurn = vi.fn(async () => ({ - sessionId: "run-1-i1", response: "capped answer", stopReason: "max_tokens", + sessionId: 'run-1-i1', + response: 'capped answer', + stopReason: 'max_tokens', usage: { input: 5, output: 9, cacheRead: 0, cacheWrite: 0, total: 14 }, })); const r = await runLeaf(base, undefined, { executeTurn }); - expect(r).toEqual({ status: "responded", text: "capped answer", usage: { input: 5, output: 9, cacheRead: 0, cacheWrite: 0, total: 14 } }); + expect(r).toEqual({ + status: 'responded', + text: 'capped answer', + usage: { input: 5, output: 9, cacheRead: 0, cacheWrite: 0, total: 14 }, + }); }); - it("maps stopReason error → failed/error carrying the message", async () => { + it('maps stopReason error → failed/error carrying the message', async () => { const executeTurn = vi.fn(async () => ({ - sessionId: "run-1-i1", response: "", stopReason: "error", errorMessage: "model exploded", + sessionId: 'run-1-i1', + response: '', + stopReason: 'error', + errorMessage: 'model exploded', })); const r = await runLeaf(base, undefined, { executeTurn }); - expect(r).toEqual({ status: "failed", reason: "error", message: "model exploded" }); + expect(r).toEqual({ status: 'failed', reason: 'error', message: 'model exploded' }); }); - it("maps stopReason aborted → aborted", async () => { - const executeTurn = vi.fn(async () => ({ sessionId: "run-1-i1", response: "", stopReason: "aborted" })); + it('maps stopReason aborted → aborted', async () => { + const executeTurn = vi.fn(async () => ({ + sessionId: 'run-1-i1', + response: '', + stopReason: 'aborted', + })); const r = await runLeaf(base, undefined, { executeTurn }); - expect(r).toEqual({ status: "aborted" }); + expect(r).toEqual({ status: 'aborted' }); }); - it("fails bad_inputs when prompt is missing", async () => { - const r = await runLeaf({ sessionId: "s", item: base.item, kind: "prompt" }, undefined, { executeTurn: vi.fn() }); - expect(r).toEqual({ status: "failed", reason: "bad_inputs" }); + it('fails bad_inputs when prompt is missing', async () => { + const r = await runLeaf({ sessionId: 's', item: base.item, kind: 'prompt' }, undefined, { + executeTurn: vi.fn(), + }); + expect(r).toEqual({ status: 'failed', reason: 'bad_inputs' }); }); }); -it("solve without env_key uses convergeWorkspace, not swebench setup", async () => { +it('solve without env_key uses convergeWorkspace, not swebench setup', async () => { // produceSolve is injectable; assert the swebench branch is NOT taken when env_key is absent. // (Structural: import isSwebenchEnvelope and check the predicate.) - const { isSwebenchEnvelope } = await import("../src/run-leaf.js"); - expect(isSwebenchEnvelope({ kind: "solve", problemStatement: "x", repoUrl: "git://h/r.git", ref: "main" })).toBe(false); - expect(isSwebenchEnvelope({ kind: "solve", problemStatement: "x", repoUrl: "/repos/a/b.git", ref: "c", env_key: "k:latest" })).toBe(true); + const { isSwebenchEnvelope } = await import('../src/run-leaf.js'); + expect( + isSwebenchEnvelope({ + kind: 'solve', + problemStatement: 'x', + repoUrl: 'git://h/r.git', + ref: 'main', + }), + ).toBe(false); + expect( + isSwebenchEnvelope({ + kind: 'solve', + problemStatement: 'x', + repoUrl: '/repos/a/b.git', + ref: 'c', + env_key: 'k:latest', + }), + ).toBe(true); }); -describe("runPromptLeaf sandbox leasing", () => { - const FAKE_CONFIG = { pod: "sandbox-0", namespace: "default", context: undefined, podCwd: "/workspace", headCwd: "/head" }; +describe('runPromptLeaf sandbox leasing', () => { + const FAKE_CONFIG = { + pod: 'sandbox-0', + namespace: 'default', + context: undefined, + podCwd: '/workspace', + headCwd: '/head', + }; const base: LeafEnvelope = { - sessionId: "run-1/i1", item: { item_id: "x", file: "f", pattern: "p" }, - kind: "prompt", prompt: "Read /etc/os-release and name the distro.", + sessionId: 'run-1/i1', + item: { item_id: 'x', file: 'f', pattern: 'p' }, + kind: 'prompt', + prompt: 'Read /etc/os-release and name the distro.', }; - const okTurn = () => vi.fn(async () => ({ sessionId: "run-1-i1", response: "RHEL 9.8", stopReason: "end_turn" })); + const okTurn = () => + vi.fn(async () => ({ sessionId: 'run-1-i1', response: 'RHEL 9.8', stopReason: 'end_turn' })); // A leased sandbox reaches the turn only if runPromptLeaf hands it over: before this, a prompt // leaf resolved its own sandbox from process.env and the lease was never consulted. - it("hands the leased grpc transport to the turn so tool calls reach the remote sandbox", async () => { - const transport = { exec: vi.fn(async () => ({ stdout: Buffer.from(""), exitCode: 0, truncated: false })), close: vi.fn(async () => {}) }; + it('hands the leased grpc transport to the turn so tool calls reach the remote sandbox', async () => { + const transport = { + exec: vi.fn(async () => ({ stdout: Buffer.from(''), exitCode: 0, truncated: false })), + close: vi.fn(async () => {}), + }; selectPoolSandboxMock.mockReset().mockResolvedValue({ - config: FAKE_CONFIG, transport, heartbeat: vi.fn(async () => {}), release: vi.fn(async () => {}), + config: FAKE_CONFIG, + transport, + heartbeat: vi.fn(async () => {}), + release: vi.fn(async () => {}), }); const executeTurn = okTurn(); await runLeaf(base, undefined, { executeTurn }); - expect(executeTurn).toHaveBeenCalledWith(expect.objectContaining({ - sandbox: { config: FAKE_CONFIG, transport }, - })); + expect(executeTurn).toHaveBeenCalledWith( + expect.objectContaining({ + sandbox: { config: FAKE_CONFIG, transport }, + }), + ); }); - it("asks for remote candidates when SH_REMOTE_SANDBOX=1", async () => { + it('asks for remote candidates when SH_REMOTE_SANDBOX=1', async () => { selectPoolSandboxMock.mockReset().mockResolvedValue({ - config: FAKE_CONFIG, heartbeat: vi.fn(async () => {}), release: vi.fn(async () => {}), + config: FAKE_CONFIG, + heartbeat: vi.fn(async () => {}), + release: vi.fn(async () => {}), }); const prev = process.env.SH_REMOTE_SANDBOX; - process.env.SH_REMOTE_SANDBOX = "1"; + process.env.SH_REMOTE_SANDBOX = '1'; try { await runLeaf(base, undefined, { executeTurn: okTurn() }); } finally { - if (prev === undefined) delete process.env.SH_REMOTE_SANDBOX; else process.env.SH_REMOTE_SANDBOX = prev; + if (prev === undefined) delete process.env.SH_REMOTE_SANDBOX; + else process.env.SH_REMOTE_SANDBOX = prev; } expect(selectPoolSandboxMock).toHaveBeenCalledWith( - expect.any(Object), expect.any(String), expect.any(String), + expect.any(Object), + expect.any(String), + expect.any(String), expect.objectContaining({ remoteSandbox: true }), ); }); - it("uses a request-scoped sandbox pool selector", async () => { + it('uses a request-scoped sandbox pool selector', async () => { selectPoolSandboxMock.mockReset().mockResolvedValue({ - config: FAKE_CONFIG, heartbeat: vi.fn(async () => {}), release: vi.fn(async () => {}), + config: FAKE_CONFIG, + heartbeat: vi.fn(async () => {}), + release: vi.fn(async () => {}), }); - await runLeaf({ ...base, sandboxPoolSelector: "sh.kagenti.io/sandbox-pool=demo-remote-only" }, - undefined, { executeTurn: okTurn() }); + await runLeaf( + { ...base, sandboxPoolSelector: 'sh.kagenti.io/sandbox-pool=demo-remote-only' }, + undefined, + { executeTurn: okTurn() }, + ); expect(selectPoolSandboxMock).toHaveBeenCalledWith( - expect.objectContaining({ KAGENTI_SANDBOX_POOL_SELECTOR: "sh.kagenti.io/sandbox-pool=demo-remote-only" }), - expect.any(String), expect.any(String), expect.any(Object), + expect.objectContaining({ + KAGENTI_SANDBOX_POOL_SELECTOR: 'sh.kagenti.io/sandbox-pool=demo-remote-only', + }), + expect.any(String), + expect.any(String), + expect.any(Object), ); }); - it("maps pool saturation to failed/saturated rather than throwing", async () => { - selectPoolSandboxMock.mockReset().mockRejectedValue(new SandboxPoolSaturatedError("pool=x")); + it('maps pool saturation to failed/saturated rather than throwing', async () => { + selectPoolSandboxMock.mockReset().mockRejectedValue(new SandboxPoolSaturatedError('pool=x')); const executeTurn = okTurn(); const r = await runLeaf(base, undefined, { executeTurn }); - expect(r).toMatchObject({ status: "failed", reason: "saturated" }); + expect(r).toMatchObject({ status: 'failed', reason: 'saturated' }); expect(executeTurn).not.toHaveBeenCalled(); }); - it("releases the lease and closes the leased transport after the turn", async () => { + it('releases the lease and closes the leased transport after the turn', async () => { const release = vi.fn(async () => {}); const close = vi.fn(async () => {}); selectPoolSandboxMock.mockReset().mockResolvedValue({ @@ -506,16 +666,20 @@ describe("runPromptLeaf sandbox leasing", () => { }); // A lease held past a crashed turn shrinks pool capacity until its TTL expires. - it("releases the lease even when the turn throws", async () => { + it('releases the lease even when the turn throws', async () => { const release = vi.fn(async () => {}); selectPoolSandboxMock.mockReset().mockResolvedValue({ - config: FAKE_CONFIG, heartbeat: vi.fn(async () => {}), release, + config: FAKE_CONFIG, + heartbeat: vi.fn(async () => {}), + release, + }); + const executeTurn = vi.fn(async () => { + throw new Error('turn exploded'); }); - const executeTurn = vi.fn(async () => { throw new Error("turn exploded"); }); const r = await runLeaf(base, undefined, { executeTurn }); - expect(r).toMatchObject({ status: "failed", reason: "error", message: "turn exploded" }); + expect(r).toMatchObject({ status: 'failed', reason: 'error', message: 'turn exploded' }); expect(release).toHaveBeenCalledTimes(1); }); }); diff --git a/harness/test/run-turn-model.test.ts b/harness/test/run-turn-model.test.ts index 83148c1..63b0e4f 100644 --- a/harness/test/run-turn-model.test.ts +++ b/harness/test/run-turn-model.test.ts @@ -1,68 +1,70 @@ -import { describe, it, expect } from "vitest"; -import { resolveModelSelection, requireModel } from "../src/run-turn"; +import { describe, it, expect } from 'vitest'; +import { resolveModelSelection, requireModel } from '../src/run-turn'; -describe("resolveModelSelection", () => { - it("defaults to anthropic / claude-opus-4-8 when nothing is set", () => { +describe('resolveModelSelection', () => { + it('defaults to anthropic / claude-opus-4-8 when nothing is set', () => { expect(resolveModelSelection(undefined, {})).toEqual({ - provider: "anthropic", - modelId: "claude-opus-4-8", + provider: 'anthropic', + modelId: 'claude-opus-4-8', }); }); - it("reads env when config is absent", () => { + it('reads env when config is absent', () => { expect( - resolveModelSelection(undefined, { SH_MODEL_PROVIDER: "openai", SH_MODEL: "gpt-x" }), - ).toEqual({ provider: "openai", modelId: "gpt-x" }); + resolveModelSelection(undefined, { SH_MODEL_PROVIDER: 'openai', SH_MODEL: 'gpt-x' }), + ).toEqual({ provider: 'openai', modelId: 'gpt-x' }); }); - it("config overrides env and defaults", () => { + it('config overrides env and defaults', () => { expect( resolveModelSelection( - { provider: "anthropic", model: "claude-sonnet-4-6" }, - { SH_MODEL_PROVIDER: "openai", SH_MODEL: "gpt-x" }, + { provider: 'anthropic', model: 'claude-sonnet-4-6' }, + { SH_MODEL_PROVIDER: 'openai', SH_MODEL: 'gpt-x' }, ), - ).toEqual({ provider: "anthropic", modelId: "claude-sonnet-4-6" }); + ).toEqual({ provider: 'anthropic', modelId: 'claude-sonnet-4-6' }); }); - it("mixes config and env per-field", () => { - expect(resolveModelSelection({ model: "m1" }, { SH_MODEL_PROVIDER: "p1" })).toEqual({ - provider: "p1", - modelId: "m1", + it('mixes config and env per-field', () => { + expect(resolveModelSelection({ model: 'm1' }, { SH_MODEL_PROVIDER: 'p1' })).toEqual({ + provider: 'p1', + modelId: 'm1', }); }); }); -describe("requireModel", () => { - it("returns the model for a known anthropic id", () => { - const m = requireModel("anthropic", "claude-opus-4-8"); - expect((m as { id: string }).id).toBe("claude-opus-4-8"); +describe('requireModel', () => { + it('returns the model for a known anthropic id', () => { + const m = requireModel('anthropic', 'claude-opus-4-8'); + expect((m as { id: string }).id).toBe('claude-opus-4-8'); }); - it("throws a clear error for a known provider but unknown model id (dot-vs-dash trap)", () => { + it('throws a clear error for a known provider but unknown model id (dot-vs-dash trap)', () => { // 'claude-sonnet-4.6' (dot) is a github-copilot key, not anthropic; the anthropic id is the dash form. - expect(() => requireModel("anthropic", "claude-sonnet-4.6")).toThrowError( + expect(() => requireModel('anthropic', 'claude-sonnet-4.6')).toThrowError( /Unknown model "anthropic\/claude-sonnet-4\.6".*claude-sonnet-4-6/s, ); }); - it("throws naming valid providers when the provider is unknown", () => { - expect(() => requireModel("litellm", "whatever")).toThrowError( + it('throws naming valid providers when the provider is unknown', () => { + expect(() => requireModel('litellm', 'whatever')).toThrowError( /Unknown model provider "litellm".*anthropic/s, ); }); }); -describe("requireModel with SH_MODEL_CUSTOM (self-hosted endpoint)", () => { - it("requires a base URL (SH_MODEL_BASE_URL or ANTHROPIC_BASE_URL) when SH_MODEL_CUSTOM=1", () => { +describe('requireModel with SH_MODEL_CUSTOM (self-hosted endpoint)', () => { + it('requires a base URL (SH_MODEL_BASE_URL or ANTHROPIC_BASE_URL) when SH_MODEL_CUSTOM=1', () => { expect(() => - requireModel("anthropic", "meta-llama/Llama-3.3-70B-Instruct", { SH_MODEL_CUSTOM: "1" }), - ).toThrowError(/SH_MODEL_CUSTOM=1 \(anthropic\) requires SH_MODEL_BASE_URL or ANTHROPIC_BASE_URL/); + requireModel('anthropic', 'meta-llama/Llama-3.3-70B-Instruct', { SH_MODEL_CUSTOM: '1' }), + ).toThrowError( + /SH_MODEL_CUSTOM=1 \(anthropic\) requires SH_MODEL_BASE_URL or ANTHROPIC_BASE_URL/, + ); }); - it("synthesizes an anthropic-messages model from SH_MODEL + ANTHROPIC_BASE_URL", () => { - const m = requireModel("anthropic", "meta-llama/Llama-3.3-70B-Instruct", { - SH_MODEL_CUSTOM: "1", - ANTHROPIC_BASE_URL: "http://vllm.my-ns.svc:8000", + it('synthesizes an anthropic-messages model from SH_MODEL + ANTHROPIC_BASE_URL', () => { + const m = requireModel('anthropic', 'meta-llama/Llama-3.3-70B-Instruct', { + SH_MODEL_CUSTOM: '1', + ANTHROPIC_BASE_URL: 'http://vllm.my-ns.svc:8000', }) as { id: string; name: string; @@ -72,48 +74,48 @@ describe("requireModel with SH_MODEL_CUSTOM (self-hosted endpoint)", () => { contextWindow: number; maxTokens: number; }; - expect(m.id).toBe("meta-llama/Llama-3.3-70B-Instruct"); - expect(m.name).toBe("meta-llama/Llama-3.3-70B-Instruct"); - expect(m.api).toBe("anthropic-messages"); - expect(m.baseUrl).toBe("http://vllm.my-ns.svc:8000"); + expect(m.id).toBe('meta-llama/Llama-3.3-70B-Instruct'); + expect(m.name).toBe('meta-llama/Llama-3.3-70B-Instruct'); + expect(m.api).toBe('anthropic-messages'); + expect(m.baseUrl).toBe('http://vllm.my-ns.svc:8000'); // provider defaults to "anthropic" so pi's key lookup resolves ANTHROPIC_API_KEY. - expect(m.provider).toBe("anthropic"); + expect(m.provider).toBe('anthropic'); // Conservative defaults when overrides are unset. expect(m.contextWindow).toBe(131072); expect(m.maxTokens).toBe(8192); }); - it("honors SH_MODEL_PROVIDER / SH_MODEL_CONTEXT_WINDOW / SH_MODEL_MAX_TOKENS overrides", () => { - const m = requireModel("anthropic", "some/model", { - SH_MODEL_CUSTOM: "1", - ANTHROPIC_BASE_URL: "http://endpoint:8000", - SH_MODEL_PROVIDER: "vllm", - SH_MODEL_CONTEXT_WINDOW: "65536", - SH_MODEL_MAX_TOKENS: "4096", + it('honors SH_MODEL_PROVIDER / SH_MODEL_CONTEXT_WINDOW / SH_MODEL_MAX_TOKENS overrides', () => { + const m = requireModel('anthropic', 'some/model', { + SH_MODEL_CUSTOM: '1', + ANTHROPIC_BASE_URL: 'http://endpoint:8000', + SH_MODEL_PROVIDER: 'vllm', + SH_MODEL_CONTEXT_WINDOW: '65536', + SH_MODEL_MAX_TOKENS: '4096', }) as { provider: string; contextWindow: number; maxTokens: number }; - expect(m.provider).toBe("vllm"); + expect(m.provider).toBe('vllm'); expect(m.contextWindow).toBe(65536); expect(m.maxTokens).toBe(4096); }); - it("defaults to the anthropic protocol when SH_MODEL_API is unset (back-compat)", () => { - const m = requireModel("anthropic", "some/model", { - SH_MODEL_CUSTOM: "1", - ANTHROPIC_BASE_URL: "http://endpoint:8000", + it('defaults to the anthropic protocol when SH_MODEL_API is unset (back-compat)', () => { + const m = requireModel('anthropic', 'some/model', { + SH_MODEL_CUSTOM: '1', + ANTHROPIC_BASE_URL: 'http://endpoint:8000', }) as { api: string; provider: string }; - expect(m.api).toBe("anthropic-messages"); - expect(m.provider).toBe("anthropic"); + expect(m.api).toBe('anthropic-messages'); + expect(m.provider).toBe('anthropic'); }); }); -describe("requireModel with SH_MODEL_API=openai-completions (OpenAI-compatible endpoints)", () => { - it("synthesizes an openai-completions model from SH_MODEL + SH_MODEL_BASE_URL + headers", () => { - const m = requireModel("openai", "moonshotai/Kimi-K2.7-Code", { - SH_MODEL_CUSTOM: "1", - SH_MODEL_API: "openai-completions", - SH_MODEL_BASE_URL: "https://rits.example/kimi/v1", +describe('requireModel with SH_MODEL_API=openai-completions (OpenAI-compatible endpoints)', () => { + it('synthesizes an openai-completions model from SH_MODEL + SH_MODEL_BASE_URL + headers', () => { + const m = requireModel('openai', 'moonshotai/Kimi-K2.7-Code', { + SH_MODEL_CUSTOM: '1', + SH_MODEL_API: 'openai-completions', + SH_MODEL_BASE_URL: 'https://rits.example/kimi/v1', SH_MODEL_HEADERS: '{"RITS_API_KEY":"abc123"}', - OPENAI_API_KEY: "present", // avoid the placeholder seed for bearer default + OPENAI_API_KEY: 'present', // avoid the placeholder seed for bearer default }) as { id: string; api: string; @@ -121,87 +123,87 @@ describe("requireModel with SH_MODEL_API=openai-completions (OpenAI-compatible e baseUrl: string; headers: Record; }; - expect(m.id).toBe("moonshotai/Kimi-K2.7-Code"); - expect(m.api).toBe("openai-completions"); + expect(m.id).toBe('moonshotai/Kimi-K2.7-Code'); + expect(m.api).toBe('openai-completions'); // provider defaults to "openai" so pi's key lookup resolves OPENAI_API_KEY. - expect(m.provider).toBe("openai"); - expect(m.baseUrl).toBe("https://rits.example/kimi/v1"); - expect(m.headers.RITS_API_KEY).toBe("abc123"); + expect(m.provider).toBe('openai'); + expect(m.baseUrl).toBe('https://rits.example/kimi/v1'); + expect(m.headers.RITS_API_KEY).toBe('abc123'); }); - it("falls back to OPENAI_BASE_URL when SH_MODEL_BASE_URL is unset", () => { - const m = requireModel("openai", "m", { - SH_MODEL_CUSTOM: "1", - SH_MODEL_API: "openai-completions", - OPENAI_BASE_URL: "https://vllm.svc/v1", - OPENAI_API_KEY: "present", + it('falls back to OPENAI_BASE_URL when SH_MODEL_BASE_URL is unset', () => { + const m = requireModel('openai', 'm', { + SH_MODEL_CUSTOM: '1', + SH_MODEL_API: 'openai-completions', + OPENAI_BASE_URL: 'https://vllm.svc/v1', + OPENAI_API_KEY: 'present', }) as { baseUrl: string }; - expect(m.baseUrl).toBe("https://vllm.svc/v1"); + expect(m.baseUrl).toBe('https://vllm.svc/v1'); }); - it("custom-header auth strips the default Authorization Bearer, keeping the custom header", () => { - const m = requireModel("openai", "m", { - SH_MODEL_CUSTOM: "1", - SH_MODEL_API: "openai-completions", - SH_MODEL_BASE_URL: "https://rits.example/v1", - SH_MODEL_AUTH: "custom-header", + it('custom-header auth strips the default Authorization Bearer, keeping the custom header', () => { + const m = requireModel('openai', 'm', { + SH_MODEL_CUSTOM: '1', + SH_MODEL_API: 'openai-completions', + SH_MODEL_BASE_URL: 'https://rits.example/v1', + SH_MODEL_AUTH: 'custom-header', SH_MODEL_HEADERS: '{"RITS_API_KEY":"abc123"}', - OPENAI_API_KEY: "present", // present ⇒ no global process.env seed side-effect in this test + OPENAI_API_KEY: 'present', // present ⇒ no global process.env seed side-effect in this test }) as { headers: Record }; expect(m.headers.Authorization).toBeNull(); - expect(m.headers.RITS_API_KEY).toBe("abc123"); + expect(m.headers.RITS_API_KEY).toBe('abc123'); }); - it("interpolates ${VAR} in header values from env (secretKeyRef indirection)", () => { - const m = requireModel("openai", "m", { - SH_MODEL_CUSTOM: "1", - SH_MODEL_API: "openai-completions", - SH_MODEL_BASE_URL: "https://rits.example/v1", - SH_MODEL_AUTH: "custom-header", + it('interpolates ${VAR} in header values from env (secretKeyRef indirection)', () => { + const m = requireModel('openai', 'm', { + SH_MODEL_CUSTOM: '1', + SH_MODEL_API: 'openai-completions', + SH_MODEL_BASE_URL: 'https://rits.example/v1', + SH_MODEL_AUTH: 'custom-header', SH_MODEL_HEADERS: '{"RITS_API_KEY":"${RITS_API_KEY}"}', - RITS_API_KEY: "secret-from-secretkeyref", - OPENAI_API_KEY: "present", + RITS_API_KEY: 'secret-from-secretkeyref', + OPENAI_API_KEY: 'present', }) as { headers: Record }; - expect(m.headers.RITS_API_KEY).toBe("secret-from-secretkeyref"); + expect(m.headers.RITS_API_KEY).toBe('secret-from-secretkeyref'); expect(m.headers.Authorization).toBeNull(); }); - it("requires SH_MODEL_BASE_URL or OPENAI_BASE_URL", () => { + it('requires SH_MODEL_BASE_URL or OPENAI_BASE_URL', () => { expect(() => - requireModel("openai", "m", { SH_MODEL_CUSTOM: "1", SH_MODEL_API: "openai-completions" }), + requireModel('openai', 'm', { SH_MODEL_CUSTOM: '1', SH_MODEL_API: 'openai-completions' }), ).toThrowError(/requires SH_MODEL_BASE_URL or OPENAI_BASE_URL/); }); - it("rejects malformed SH_MODEL_HEADERS", () => { + it('rejects malformed SH_MODEL_HEADERS', () => { expect(() => - requireModel("openai", "m", { - SH_MODEL_CUSTOM: "1", - SH_MODEL_API: "openai-completions", - SH_MODEL_BASE_URL: "https://x/v1", - SH_MODEL_HEADERS: "not-json", - OPENAI_API_KEY: "present", + requireModel('openai', 'm', { + SH_MODEL_CUSTOM: '1', + SH_MODEL_API: 'openai-completions', + SH_MODEL_BASE_URL: 'https://x/v1', + SH_MODEL_HEADERS: 'not-json', + OPENAI_API_KEY: 'present', }), ).toThrowError(/SH_MODEL_HEADERS must be a JSON object/); }); }); -describe("requireModel SH_MODEL_API validation", () => { +describe('requireModel SH_MODEL_API validation', () => { it("throws 'not yet implemented' for openai-responses (deferred)", () => { expect(() => - requireModel("openai", "m", { - SH_MODEL_CUSTOM: "1", - SH_MODEL_API: "openai-responses", - SH_MODEL_BASE_URL: "https://x/v1", + requireModel('openai', 'm', { + SH_MODEL_CUSTOM: '1', + SH_MODEL_API: 'openai-responses', + SH_MODEL_BASE_URL: 'https://x/v1', }), ).toThrowError(/openai-responses is not yet implemented/); }); - it("throws a clear error for an unknown SH_MODEL_API", () => { + it('throws a clear error for an unknown SH_MODEL_API', () => { expect(() => - requireModel("openai", "m", { - SH_MODEL_CUSTOM: "1", - SH_MODEL_API: "grpc-magic", - SH_MODEL_BASE_URL: "https://x/v1", + requireModel('openai', 'm', { + SH_MODEL_CUSTOM: '1', + SH_MODEL_API: 'grpc-magic', + SH_MODEL_BASE_URL: 'https://x/v1', }), ).toThrowError(/Unknown SH_MODEL_API "grpc-magic"/); }); diff --git a/harness/test/run-turn-sandbox.test.ts b/harness/test/run-turn-sandbox.test.ts index 6380cb0..6b21b7e 100644 --- a/harness/test/run-turn-sandbox.test.ts +++ b/harness/test/run-turn-sandbox.test.ts @@ -1,40 +1,53 @@ -import { describe, it, expect } from "vitest"; -import { resolveTurnSandbox } from "../src/run-turn.js"; +import { describe, it, expect } from 'vitest'; +import { resolveTurnSandbox } from '../src/run-turn.js'; // executeTurn resolved its sandbox from process.env unconditionally (ADR 0028: prompt leaves // "inherit /turn's sandbox routing"), which left a leased pool sandbox — and the whole remote // relay transport — unreachable from a prompt leaf. resolveTurnSandbox is the seam: a caller // that has already leased a sandbox injects it; /turn injects nothing and keeps env resolution. -describe("resolveTurnSandbox", () => { - it("returns the injected sandbox verbatim, including its transport", async () => { - const transport = { exec: async () => ({ stdout: Buffer.from(""), exitCode: 0, truncated: false }), close: async () => {} }; - const injected = { config: { pod: "sbx-laptop", namespace: "default", podCwd: "/workspace", headCwd: "/head" }, transport }; - - const got = await resolveTurnSandbox(injected, {}, "/head"); +describe('resolveTurnSandbox', () => { + it('returns the injected sandbox verbatim, including its transport', async () => { + const transport = { + exec: async () => ({ stdout: Buffer.from(''), exitCode: 0, truncated: false }), + close: async () => {}, + }; + const injected = { + config: { pod: 'sbx-laptop', namespace: 'default', podCwd: '/workspace', headCwd: '/head' }, + transport, + }; + + const got = await resolveTurnSandbox(injected, {}, '/head'); expect(got).toEqual(injected); expect(got.transport).toBe(transport); }); - it("ignores env resolution entirely when a sandbox is injected", async () => { + it('ignores env resolution entirely when a sandbox is injected', async () => { // A leased remote sandbox must win over an ambient KAGENTI_SANDBOX_POD: honouring the env // here would silently route a remote prompt leaf back to an in-cluster pod. - const injected = { config: { pod: "sbx-leased", namespace: "default", podCwd: "/workspace", headCwd: "/head" } }; + const injected = { + config: { pod: 'sbx-leased', namespace: 'default', podCwd: '/workspace', headCwd: '/head' }, + }; - const got = await resolveTurnSandbox(injected, { KAGENTI_SANDBOX_POD: "sandbox-0" }, "/head"); + const got = await resolveTurnSandbox(injected, { KAGENTI_SANDBOX_POD: 'sandbox-0' }, '/head'); - expect(got.config?.pod).toBe("sbx-leased"); + expect(got.config?.pod).toBe('sbx-leased'); }); - it("falls back to env resolution when nothing is injected (the /turn path)", async () => { - const got = await resolveTurnSandbox(undefined, { KAGENTI_SANDBOX_POD: "sandbox-0" }, "/head"); + it('falls back to env resolution when nothing is injected (the /turn path)', async () => { + const got = await resolveTurnSandbox(undefined, { KAGENTI_SANDBOX_POD: 'sandbox-0' }, '/head'); - expect(got.config).toMatchObject({ pod: "sandbox-0", namespace: "default", podCwd: "/workspace", headCwd: "/head" }); + expect(got.config).toMatchObject({ + pod: 'sandbox-0', + namespace: 'default', + podCwd: '/workspace', + headCwd: '/head', + }); expect(got.transport).toBeUndefined(); }); - it("resolves to a null config when neither injected nor configured (tools run local)", async () => { - const got = await resolveTurnSandbox(undefined, {}, "/head"); + it('resolves to a null config when neither injected nor configured (tools run local)', async () => { + const got = await resolveTurnSandbox(undefined, {}, '/head'); expect(got).toEqual({ config: null }); }); diff --git a/harness/test/run-turn.test.ts b/harness/test/run-turn.test.ts index 2f70bc2..7b4dd01 100644 --- a/harness/test/run-turn.test.ts +++ b/harness/test/run-turn.test.ts @@ -1,15 +1,15 @@ -import { describe, it, expect, afterAll } from "vitest"; -import { RedisSessionBackend } from "@sh/session-backend"; -import type { FileEntry } from "@earendil-works/pi-coding-agent"; -import { runTurn, executeTurn, wireAbort } from "../src/run-turn.js"; +import { describe, it, expect, afterAll } from 'vitest'; +import { RedisSessionBackend } from '@sh/session-backend'; +import type { FileEntry } from '@earendil-works/pi-coding-agent'; +import { runTurn, executeTurn, wireAbort } from '../src/run-turn.js'; -const REDIS = process.env.REDIS_URL ?? "redis://127.0.0.1:6379"; +const REDIS = process.env.REDIS_URL ?? 'redis://127.0.0.1:6379'; const store = new RedisSessionBackend(REDIS); const createdSessions: string[] = []; // These cases call the real model through the gateway. Gate them so CI does not fail on an // ungated live call (e.g. stopReason variance). Run with SH_RUN_LIVE=1 + ANTHROPIC_AUTH_TOKEN. -const LIVE = process.env.SH_RUN_LIVE === "1" && !!process.env.ANTHROPIC_AUTH_TOKEN; +const LIVE = process.env.SH_RUN_LIVE === '1' && !!process.env.ANTHROPIC_AUTH_TOKEN; afterAll(async () => { for (const sid of createdSessions) { @@ -18,73 +18,79 @@ afterAll(async () => { await store.close(); }); -describe("runTurn()", () => { - it.runIf(LIVE)("creates a new session when sessionId is undefined", async () => { - const result = await runTurn("Say exactly: PONG", undefined, { +describe('runTurn()', () => { + it.runIf(LIVE)('creates a new session when sessionId is undefined', async () => { + const result = await runTurn('Say exactly: PONG', undefined, { redisUrl: REDIS, }); expect(result.sessionId).toBeTruthy(); - expect(result.response).toContain("PONG"); - expect(result.stopReason).toBe("end_turn"); + expect(result.response).toContain('PONG'); + expect(result.stopReason).toBe('end_turn'); createdSessions.push(result.sessionId); }); - it.runIf(LIVE)("resumes an existing session from Redis", async () => { + it.runIf(LIVE)('resumes an existing session from Redis', async () => { // Create a session first - const first = await runTurn("Remember the code word: ZEBRA42", undefined, { + const first = await runTurn('Remember the code word: ZEBRA42', undefined, { redisUrl: REDIS, }); createdSessions.push(first.sessionId); // Resume and ask for recall - const second = await runTurn( - "What was the code word I told you?", - first.sessionId, - { redisUrl: REDIS }, - ); + const second = await runTurn('What was the code word I told you?', first.sessionId, { + redisUrl: REDIS, + }); expect(second.sessionId).toBe(first.sessionId); - expect(second.response).toContain("ZEBRA42"); + expect(second.response).toContain('ZEBRA42'); }); - it("throws when sessionId does not exist in Redis", async () => { + it('throws when sessionId does not exist in Redis', async () => { await expect( - runTurn("hello", "nonexistent-session-id-12345", { redisUrl: REDIS }), - ).rejects.toThrow("no session in backend"); + runTurn('hello', 'nonexistent-session-id-12345', { redisUrl: REDIS }), + ).rejects.toThrow('no session in backend'); }); }); -describe("executeTurn / runTurn 404 contract", () => { - it("exposes executeTurn as the shared core", () => { - expect(typeof executeTurn).toBe("function"); +describe('executeTurn / runTurn 404 contract', () => { + it('exposes executeTurn as the shared core', () => { + expect(typeof executeTurn).toBe('function'); }); - it("executeTurn with createIfAbsent:false throws when the session is absent", async () => { + it('executeTurn with createIfAbsent:false throws when the session is absent', async () => { await expect( executeTurn({ - prompt: "hello", - sessionId: "nonexistent-session-id-98765", + prompt: 'hello', + sessionId: 'nonexistent-session-id-98765', config: { redisUrl: REDIS }, createIfAbsent: false, }), - ).rejects.toThrow("no session in backend"); + ).rejects.toThrow('no session in backend'); }); }); -describe("wireAbort", () => { - it("calls session.abort() immediately when the signal is already aborted", () => { +describe('wireAbort', () => { + it('calls session.abort() immediately when the signal is already aborted', () => { let aborted = 0; const ac = new AbortController(); ac.abort(); - wireAbort(ac.signal, { abort: () => { aborted++; } }); + wireAbort(ac.signal, { + abort: () => { + aborted++; + }, + }); expect(aborted).toBe(1); }); - it("calls session.abort() once when the signal fires later, and is idempotent", () => { + it('calls session.abort() once when the signal fires later, and is idempotent', () => { let aborted = 0; const ac = new AbortController(); - wireAbort(ac.signal, { abort: () => { aborted++; } }); + wireAbort(ac.signal, { + abort: () => { + aborted++; + }, + }); expect(aborted).toBe(0); ac.abort(); expect(aborted).toBe(1); diff --git a/harness/test/sandbox-lease.test.ts b/harness/test/sandbox-lease.test.ts index 221e53e..29191f6 100644 --- a/harness/test/sandbox-lease.test.ts +++ b/harness/test/sandbox-lease.test.ts @@ -1,24 +1,24 @@ -import { describe, it, expect } from "vitest"; -import { leaseKey, activeCount } from "../src/sandbox-lease.js"; +import { describe, it, expect } from 'vitest'; +import { leaseKey, activeCount } from '../src/sandbox-lease.js'; -describe("leaseKey", () => { - it("namespaces per pod", () => { - expect(leaseKey("sandbox-1-0")).toBe("sh:sandbox:sandbox-1-0:leases"); +describe('leaseKey', () => { + it('namespaces per pod', () => { + expect(leaseKey('sandbox-1-0')).toBe('sh:sandbox:sandbox-1-0:leases'); }); }); -describe("activeCount", () => { +describe('activeCount', () => { const now = 1_000_000; - it("counts only members whose expiry is strictly in the future", () => { + it('counts only members whose expiry is strictly in the future', () => { const members = [ - { value: "a", score: now - 1 }, // expired - { value: "b", score: now }, // expired (boundary: not > now) - { value: "c", score: now + 1 }, // active - { value: "d", score: now + 500 }, // active + { value: 'a', score: now - 1 }, // expired + { value: 'b', score: now }, // expired (boundary: not > now) + { value: 'c', score: now + 1 }, // active + { value: 'd', score: now + 500 }, // active ]; expect(activeCount(members, now)).toBe(2); }); - it("is 0 for an empty set", () => { + it('is 0 for an empty set', () => { expect(activeCount([], now)).toBe(0); }); }); diff --git a/harness/test/select-sandbox.test.ts b/harness/test/select-sandbox.test.ts index 85588d2..dfa3a1e 100644 --- a/harness/test/select-sandbox.test.ts +++ b/harness/test/select-sandbox.test.ts @@ -1,8 +1,12 @@ -import { describe, it, expect, vi } from "vitest"; -import { orderByLoad, selectPoolSandbox, SandboxPoolSaturatedError } from "../src/select-sandbox.js"; -import type { LeaseStore } from "../src/sandbox-lease.js"; -import type { RecordStore, SandboxRecord } from "../src/pool-records.js"; -import type { ExecClientLike } from "@sh/k8s-sandbox"; +import { describe, it, expect, vi } from 'vitest'; +import { + orderByLoad, + selectPoolSandbox, + SandboxPoolSaturatedError, +} from '../src/select-sandbox.js'; +import type { LeaseStore } from '../src/sandbox-lease.js'; +import type { RecordStore, SandboxRecord } from '../src/pool-records.js'; +import type { ExecClientLike } from '@sh/k8s-sandbox'; // Spy on the RedisRecordStore constructor select-sandbox.ts falls back to when // deps.records isn't injected, so we can assert its lifecycle (list + close) @@ -10,8 +14,8 @@ import type { ExecClientLike } from "@sh/k8s-sandbox"; const { createdRecordStores } = vi.hoisted(() => ({ createdRecordStores: [] as { list: ReturnType; close: ReturnType }[], })); -vi.mock("../src/pool-records.js", async (importOriginal) => { - const actual = await importOriginal(); +vi.mock('../src/pool-records.js', async (importOriginal) => { + const actual = await importOriginal(); return { ...actual, RedisRecordStore: vi.fn().mockImplementation(() => { @@ -27,24 +31,38 @@ vi.mock("../src/pool-records.js", async (importOriginal) => { }; }); -describe("orderByLoad", () => { - it("sorts ascending by active load, stable on ties", () => { - expect(orderByLoad([ - { pod: "c", active: 2 }, { pod: "a", active: 0 }, { pod: "b", active: 0 }, - ])).toEqual(["a", "b", "c"]); +describe('orderByLoad', () => { + it('sorts ascending by active load, stable on ties', () => { + expect( + orderByLoad([ + { pod: 'c', active: 2 }, + { pod: 'a', active: 0 }, + { pod: 'b', active: 0 }, + ]), + ).toEqual(['a', 'b', 'c']); }); }); -function fakeLease(loads: Record, cap: number): LeaseStore & { acquired: string[]; acquiredTtl: number[] } { +function fakeLease( + loads: Record, + cap: number, +): LeaseStore & { acquired: string[]; acquiredTtl: number[] } { const counts = { ...loads }; const acquired: string[] = []; const acquiredTtl: number[] = []; return { acquired, acquiredTtl, - async load(pod) { return counts[pod] ?? 0; }, + async load(pod) { + return counts[pod] ?? 0; + }, async acquire(pod, c, _runId, ttlMs) { - if ((counts[pod] ?? 0) < c) { counts[pod] = (counts[pod] ?? 0) + 1; acquired.push(pod); acquiredTtl.push(ttlMs); return true; } + if ((counts[pod] ?? 0) < c) { + counts[pod] = (counts[pod] ?? 0) + 1; + acquired.push(pod); + acquiredTtl.push(ttlMs); + return true; + } return false; }, async heartbeat() {}, @@ -52,47 +70,49 @@ function fakeLease(loads: Record, cap: number): LeaseStore & { a }; } -describe("selectPoolSandbox", () => { +describe('selectPoolSandbox', () => { const opts = { cap: 2, ttlMs: 60000 }; - it("returns null when no sandbox is configured at all", async () => { - const res = await selectPoolSandbox({} as NodeJS.ProcessEnv, "/head", "leaf-1", opts, { + it('returns null when no sandbox is configured at all', async () => { + const res = await selectPoolSandbox({} as NodeJS.ProcessEnv, '/head', 'leaf-1', opts, { listPods: async () => [], }); expect(res).toBeNull(); }); - it("falls back to single-pod resolution when no pool selector is set", async () => { - const env = { KAGENTI_SANDBOX_POD: "sandbox-x" } as unknown as NodeJS.ProcessEnv; - const res = await selectPoolSandbox(env, "/head", "leaf-1", opts, {}); - expect(res?.config.pod).toBe("sandbox-x"); + it('falls back to single-pod resolution when no pool selector is set', async () => { + const env = { KAGENTI_SANDBOX_POD: 'sandbox-x' } as unknown as NodeJS.ProcessEnv; + const res = await selectPoolSandbox(env, '/head', 'leaf-1', opts, {}); + expect(res?.config.pod).toBe('sandbox-x'); }); - it("picks the least-loaded pod and acquires a lease", async () => { - const env = { KAGENTI_SANDBOX_POOL_SELECTOR: "app=sandbox" } as unknown as NodeJS.ProcessEnv; - const lease = fakeLease({ "sandbox-0-0": 2, "sandbox-1-0": 0 }, opts.cap); - const res = await selectPoolSandbox(env, "/head", "leaf-1", opts, { - listPods: async () => ["sandbox-0-0", "sandbox-1-0"], + it('picks the least-loaded pod and acquires a lease', async () => { + const env = { KAGENTI_SANDBOX_POOL_SELECTOR: 'app=sandbox' } as unknown as NodeJS.ProcessEnv; + const lease = fakeLease({ 'sandbox-0-0': 2, 'sandbox-1-0': 0 }, opts.cap); + const res = await selectPoolSandbox(env, '/head', 'leaf-1', opts, { + listPods: async () => ['sandbox-0-0', 'sandbox-1-0'], lease, }); - expect(res?.config.pod).toBe("sandbox-1-0"); - expect(lease.acquired).toEqual(["sandbox-1-0"]); + expect(res?.config.pod).toBe('sandbox-1-0'); + expect(lease.acquired).toEqual(['sandbox-1-0']); expect(lease.acquiredTtl).toEqual([opts.ttlMs]); }); - it("throws SandboxPoolSaturatedError when every pod is at cap", async () => { - const env = { KAGENTI_SANDBOX_POOL_SELECTOR: "app=sandbox" } as unknown as NodeJS.ProcessEnv; - const lease = fakeLease({ "sandbox-0-0": 2, "sandbox-1-0": 2 }, opts.cap); - await expect(selectPoolSandbox(env, "/head", "leaf-1", opts, { - listPods: async () => ["sandbox-0-0", "sandbox-1-0"], - lease, - })).rejects.toBeInstanceOf(SandboxPoolSaturatedError); + it('throws SandboxPoolSaturatedError when every pod is at cap', async () => { + const env = { KAGENTI_SANDBOX_POOL_SELECTOR: 'app=sandbox' } as unknown as NodeJS.ProcessEnv; + const lease = fakeLease({ 'sandbox-0-0': 2, 'sandbox-1-0': 2 }, opts.cap); + await expect( + selectPoolSandbox(env, '/head', 'leaf-1', opts, { + listPods: async () => ['sandbox-0-0', 'sandbox-1-0'], + lease, + }), + ).rejects.toBeInstanceOf(SandboxPoolSaturatedError); }); - it("throws a plain Error (not SandboxPoolSaturatedError) when a pool selector is set but no pods are Running", async () => { - const env = { KAGENTI_SANDBOX_POOL_SELECTOR: "app=sandbox" } as unknown as NodeJS.ProcessEnv; + it('throws a plain Error (not SandboxPoolSaturatedError) when a pool selector is set but no pods are Running', async () => { + const env = { KAGENTI_SANDBOX_POOL_SELECTOR: 'app=sandbox' } as unknown as NodeJS.ProcessEnv; // lease is never touched: the empty-list guard throws before any lease call. - const err = await selectPoolSandbox(env, "/head", "leaf-1", opts, { + const err = await selectPoolSandbox(env, '/head', 'leaf-1', opts, { listPods: async () => [], lease: fakeLease({}, opts.cap), }).catch((e) => e); @@ -102,53 +122,77 @@ describe("selectPoolSandbox", () => { }); }); -const grpcRec: SandboxRecord = { sandboxId: "sbx-remote-1", labels: {}, capabilities: [], capacityMax: 4, transport: "grpc" }; -const fakeRecords = (recs: SandboxRecord[]): RecordStore => ({ put: async () => {}, remove: async () => {}, list: async () => recs }); -const fakeExecClient: ExecClientLike = { exec: () => ({ on: () => ({}), cancel: () => {} }) as never, abort: (_r, cb) => { cb(null); return {}; } }; - -describe("selectPoolSandbox remote dispatch", () => { - const env = (extra: Record = {}) => ({ KAGENTI_SANDBOX_POOL_SELECTOR: "app=sbx", ...extra }) as NodeJS.ProcessEnv; +const grpcRec: SandboxRecord = { + sandboxId: 'sbx-remote-1', + labels: {}, + capabilities: [], + capacityMax: 4, + transport: 'grpc', +}; +const fakeRecords = (recs: SandboxRecord[]): RecordStore => ({ + put: async () => {}, + remove: async () => {}, + list: async () => recs, +}); +const fakeExecClient: ExecClientLike = { + exec: () => ({ on: () => ({}), cancel: () => {} }) as never, + abort: (_r, cb) => { + cb(null); + return {}; + }, +}; + +describe('selectPoolSandbox remote dispatch', () => { + const env = (extra: Record = {}) => + ({ KAGENTI_SANDBOX_POOL_SELECTOR: 'app=sbx', ...extra }) as NodeJS.ProcessEnv; const opts = { cap: 4, ttlMs: 60000, remoteSandbox: true }; - it("flag OFF: ignores grpc records, transport is undefined, and never calls records.list()", async () => { - const lease = fakeLease({ "sandbox-0-0": 0 }, opts.cap); + it('flag OFF: ignores grpc records, transport is undefined, and never calls records.list()', async () => { + const lease = fakeLease({ 'sandbox-0-0': 0 }, opts.cap); const list = vi.fn(async () => [grpcRec]); const records: RecordStore = { put: async () => {}, remove: async () => {}, list }; - const sel = await selectPoolSandbox(env(), "/head", "run-1", { cap: 4, ttlMs: 60000 /* remoteSandbox omitted ⇒ false */ }, { - listPods: async () => ["sandbox-0-0"], - lease, - records, - }); + const sel = await selectPoolSandbox( + env(), + '/head', + 'run-1', + { cap: 4, ttlMs: 60000 /* remoteSandbox omitted ⇒ false */ }, + { + listPods: async () => ['sandbox-0-0'], + lease, + records, + }, + ); expect(sel?.transport).toBeUndefined(); - expect(sel?.config.pod).toBe("sandbox-0-0"); + expect(sel?.config.pod).toBe('sandbox-0-0'); // The #1 inertness gate: when remoteSandbox is off, RecordStore.list() must never be invoked. expect(list).not.toHaveBeenCalled(); }); - it("flag ON: a leased grpc record yields a GrpcRelayTransport", async () => { + it('flag ON: a leased grpc record yields a GrpcRelayTransport', async () => { // Only the grpc record is available (no pods) so it must be chosen. - const lease = fakeLease({ "sbx-remote-1": 0 }, opts.cap); - const sel = await selectPoolSandbox(env(), "/head", "run-1", opts, { + const lease = fakeLease({ 'sbx-remote-1': 0 }, opts.cap); + const sel = await selectPoolSandbox(env(), '/head', 'run-1', opts, { listPods: async () => [], lease, records: fakeRecords([grpcRec]), makeExecClient: () => fakeExecClient, }); expect(sel?.transport).toBeDefined(); - expect(typeof sel?.transport?.exec).toBe("function"); - expect(sel?.config.pod).toBe("sbx-remote-1"); + expect(typeof sel?.transport?.exec).toBe('function'); + expect(sel?.config.pod).toBe('sbx-remote-1'); }); }); -describe("selectPoolSandbox remote dispatch: ad-hoc RedisRecordStore lifecycle", () => { - const env = (extra: Record = {}) => ({ KAGENTI_SANDBOX_POOL_SELECTOR: "app=sbx", ...extra }) as NodeJS.ProcessEnv; +describe('selectPoolSandbox remote dispatch: ad-hoc RedisRecordStore lifecycle', () => { + const env = (extra: Record = {}) => + ({ KAGENTI_SANDBOX_POOL_SELECTOR: 'app=sbx', ...extra }) as NodeJS.ProcessEnv; const opts = { cap: 4, ttlMs: 60000, remoteSandbox: true }; - it("closes the RedisRecordStore it constructs itself (no deps.records injected)", async () => { + it('closes the RedisRecordStore it constructs itself (no deps.records injected)', async () => { createdRecordStores.length = 0; - const lease = fakeLease({ "sandbox-0-0": 0 }, opts.cap); - await selectPoolSandbox(env(), "/head", "run-1", opts, { - listPods: async () => ["sandbox-0-0"], + const lease = fakeLease({ 'sandbox-0-0': 0 }, opts.cap); + await selectPoolSandbox(env(), '/head', 'run-1', opts, { + listPods: async () => ['sandbox-0-0'], lease, // deps.records intentionally omitted: this exercises the not-injected branch. }); @@ -157,12 +201,12 @@ describe("selectPoolSandbox remote dispatch: ad-hoc RedisRecordStore lifecycle", expect(createdRecordStores[0].close).toHaveBeenCalledTimes(1); }); - it("does not construct (or close) a RedisRecordStore when deps.records is injected", async () => { + it('does not construct (or close) a RedisRecordStore when deps.records is injected', async () => { createdRecordStores.length = 0; - const lease = fakeLease({ "sandbox-0-0": 0 }, opts.cap); + const lease = fakeLease({ 'sandbox-0-0': 0 }, opts.cap); const injected = fakeRecords([]); - await selectPoolSandbox(env(), "/head", "run-1", opts, { - listPods: async () => ["sandbox-0-0"], + await selectPoolSandbox(env(), '/head', 'run-1', opts, { + listPods: async () => ['sandbox-0-0'], lease, records: injected, }); diff --git a/harness/test/submit-verdict-tool.test.ts b/harness/test/submit-verdict-tool.test.ts index 3785d0c..a8c2164 100644 --- a/harness/test/submit-verdict-tool.test.ts +++ b/harness/test/submit-verdict-tool.test.ts @@ -1,5 +1,5 @@ -import { describe, it, expect } from "vitest"; -import { submitVerdictExtension, type VerdictCapture } from "../src/submit-verdict-tool"; +import { describe, it, expect } from 'vitest'; +import { submitVerdictExtension, type VerdictCapture } from '../src/submit-verdict-tool'; // Minimal fake ExtensionAPI that records the registered tool. function fakePi() { @@ -7,65 +7,113 @@ function fakePi() { return { api: { registerTool: (t: any) => tools.push(t), on: () => {} } as any, tools }; } -describe("submitVerdictExtension", () => { - it("registers a submit_verdict tool", () => { +describe('submitVerdictExtension', () => { + it('registers a submit_verdict tool', () => { const capture: VerdictCapture = {}; const { api, tools } = fakePi(); submitVerdictExtension(capture)(api); expect(tools).toHaveLength(1); - expect(tools[0].name).toBe("submit_verdict"); + expect(tools[0].name).toBe('submit_verdict'); }); - it("captures a valid verdict and returns success", async () => { + it('captures a valid verdict and returns success', async () => { const capture: VerdictCapture = {}; const { api, tools } = fakePi(); submitVerdictExtension(capture)(api); - const res = await tools[0].execute("call-1", { item_id: "i1", verdict: "FLAGGED", reason: "r" }, undefined, undefined, {} as any); - expect(capture.verdict).toEqual({ item_id: "i1", verdict: "FLAGGED", reason: "r" }); + const res = await tools[0].execute( + 'call-1', + { item_id: 'i1', verdict: 'FLAGGED', reason: 'r' }, + undefined, + undefined, + {} as any, + ); + expect(capture.verdict).toEqual({ item_id: 'i1', verdict: 'FLAGGED', reason: 'r' }); expect(res.isError).toBeFalsy(); }); - it("sets terminate: true on a successful verdict, so agent-loop.ts stops the turn loop", async () => { + it('sets terminate: true on a successful verdict, so agent-loop.ts stops the turn loop', async () => { const capture: VerdictCapture = {}; const { api, tools } = fakePi(); submitVerdictExtension(capture)(api); - const res = await tools[0].execute("call-1", { item_id: "i1", verdict: "CLEAR", reason: "r" }, undefined, undefined, {} as any); + const res = await tools[0].execute( + 'call-1', + { item_id: 'i1', verdict: 'CLEAR', reason: 'r' }, + undefined, + undefined, + {} as any, + ); expect(res.terminate).toBe(true); }); - it("does not set terminate on an invalid verdict (agent should retry, not stop)", async () => { + it('does not set terminate on an invalid verdict (agent should retry, not stop)', async () => { const capture: VerdictCapture = {}; const { api, tools } = fakePi(); submitVerdictExtension(capture)(api); - const res = await tools[0].execute("call-1", { item_id: "i1", verdict: "MAYBE", reason: "r" }, undefined, undefined, {} as any); + const res = await tools[0].execute( + 'call-1', + { item_id: 'i1', verdict: 'MAYBE', reason: 'r' }, + undefined, + undefined, + {} as any, + ); expect(res.terminate).toBeFalsy(); }); - it("rejects an invalid verdict and does not capture", async () => { + it('rejects an invalid verdict and does not capture', async () => { const capture: VerdictCapture = {}; const { api, tools } = fakePi(); submitVerdictExtension(capture)(api); - const res = await tools[0].execute("call-1", { item_id: "i1", verdict: "MAYBE", reason: "r" }, undefined, undefined, {} as any); + const res = await tools[0].execute( + 'call-1', + { item_id: 'i1', verdict: 'MAYBE', reason: 'r' }, + undefined, + undefined, + {} as any, + ); expect(capture.verdict).toBeUndefined(); expect(res.isError).toBe(true); }); - it("appends a durable verdict custom entry when a session manager is provided", async () => { + it('appends a durable verdict custom entry when a session manager is provided', async () => { const capture: VerdictCapture = {}; const appended: Array<{ t: string; d: unknown }> = []; - const sm = { appendCustomEntry: (t: string, d?: unknown) => { appended.push({ t, d }); return "id"; } }; + const sm = { + appendCustomEntry: (t: string, d?: unknown) => { + appended.push({ t, d }); + return 'id'; + }, + }; const { api, tools } = fakePi(); submitVerdictExtension(capture, sm)(api); - await tools[0].execute("c1", { item_id: "i1", verdict: "CLEAR", reason: "r" }, undefined, undefined, {} as any); - expect(appended).toEqual([{ t: "verdict", d: { item_id: "i1", verdict: "CLEAR", reason: "r" } }]); + await tools[0].execute( + 'c1', + { item_id: 'i1', verdict: 'CLEAR', reason: 'r' }, + undefined, + undefined, + {} as any, + ); + expect(appended).toEqual([ + { t: 'verdict', d: { item_id: 'i1', verdict: 'CLEAR', reason: 'r' } }, + ]); }); - it("does not append a durable entry on an invalid verdict", async () => { + it('does not append a durable entry on an invalid verdict', async () => { const appended: Array<{ t: string; d: unknown }> = []; - const sm = { appendCustomEntry: (t: string, d?: unknown) => { appended.push({ t, d }); return "id"; } }; + const sm = { + appendCustomEntry: (t: string, d?: unknown) => { + appended.push({ t, d }); + return 'id'; + }, + }; const { api, tools } = fakePi(); submitVerdictExtension({}, sm)(api); - await tools[0].execute("c1", { item_id: "i1", verdict: "MAYBE", reason: "r" }, undefined, undefined, {} as any); + await tools[0].execute( + 'c1', + { item_id: 'i1', verdict: 'MAYBE', reason: 'r' }, + undefined, + undefined, + {} as any, + ); expect(appended).toEqual([]); }); }); diff --git a/harness/test/swebench-setup.test.ts b/harness/test/swebench-setup.test.ts index be9f9b3..3ee7d78 100644 --- a/harness/test/swebench-setup.test.ts +++ b/harness/test/swebench-setup.test.ts @@ -1,98 +1,123 @@ -import { describe, it, expect } from "vitest"; +import { describe, it, expect } from 'vitest'; import { - envDirFromKey, buildSwebenchSetupScript, buildSwebenchDiffScript, - swebenchCheckoutDir, buildSwebenchSolvePrompt, - setupSwebenchWorkspace, captureSwebenchDiff, -} from "../src/swebench-setup.js"; + envDirFromKey, + buildSwebenchSetupScript, + buildSwebenchDiffScript, + swebenchCheckoutDir, + buildSwebenchSolvePrompt, + setupSwebenchWorkspace, + captureSwebenchDiff, +} from '../src/swebench-setup.js'; -describe("swebench-setup script builders", () => { - const a = { repoUrl: "/repos/django/django.git", baseCommit: "abc1234", envKey: "sweb.env.py.x86_64.deadbeef:latest", runId: "run-1" }; - it("derives env_dir by stripping a trailing :latest", () => { - expect(envDirFromKey("sweb.env.py.x86_64.deadbeef:latest")).toBe("sweb.env.py.x86_64.deadbeef"); - expect(envDirFromKey("sweb.env.py.x86_64.deadbeef")).toBe("sweb.env.py.x86_64.deadbeef"); +describe('swebench-setup script builders', () => { + const a = { + repoUrl: '/repos/django/django.git', + baseCommit: 'abc1234', + envKey: 'sweb.env.py.x86_64.deadbeef:latest', + runId: 'run-1', + }; + it('derives env_dir by stripping a trailing :latest', () => { + expect(envDirFromKey('sweb.env.py.x86_64.deadbeef:latest')).toBe('sweb.env.py.x86_64.deadbeef'); + expect(envDirFromKey('sweb.env.py.x86_64.deadbeef')).toBe('sweb.env.py.x86_64.deadbeef'); }); - it("clones with --no-hardlinks, checks out base_commit, builds a system-site venv, editable-installs with build-iso fallback under HOME=/workspace", () => { + it('clones with --no-hardlinks, checks out base_commit, builds a system-site venv, editable-installs with build-iso fallback under HOME=/workspace', () => { const s = buildSwebenchSetupScript(a); expect(s).toContain("git clone --no-hardlinks '/repos/django/django.git'"); expect(s).toContain("checkout -q 'abc1234'"); - expect(s).toContain("/opt/miniconda3/envs/sweb.env.py.x86_64.deadbeef/bin/python' -m venv --system-site-packages"); - expect(s).toContain("HOME=/workspace"); - expect(s).toContain("--no-build-isolation"); - expect(s).toContain("--no-cache-dir"); + expect(s).toContain( + "/opt/miniconda3/envs/sweb.env.py.x86_64.deadbeef/bin/python' -m venv --system-site-packages", + ); + expect(s).toContain('HOME=/workspace'); + expect(s).toContain('--no-build-isolation'); + expect(s).toContain('--no-cache-dir'); // fallback: a second pip install WITHOUT --no-build-isolation expect(s.match(/pip" install -e/g)?.length).toBeGreaterThanOrEqual(2); // prints the checkout dir on stdout so the caller can set podCwd - expect(s).toContain(swebenchCheckoutDir("run-1")); + expect(s).toContain(swebenchCheckoutDir('run-1')); }); - it("diff script stages all and prints the cached diff from the checkout dir", () => { - const s = buildSwebenchDiffScript("run-1"); - expect(s).toContain(`git -C '${swebenchCheckoutDir("run-1")}' add -A`); - expect(s).toContain("diff --cached"); + it('diff script stages all and prints the cached diff from the checkout dir', () => { + const s = buildSwebenchDiffScript('run-1'); + expect(s).toContain(`git -C '${swebenchCheckoutDir('run-1')}' add -A`); + expect(s).toContain('diff --cached'); }); - it("solve prompt names the checkout root and the venv python", () => { - const p = buildSwebenchSolvePrompt("fix the bug", "/workspace/co-run-1", "/workspace/venv-run-1/bin/python"); - expect(p).toContain("/workspace/co-run-1"); - expect(p).toContain("/workspace/venv-run-1/bin/python"); - expect(p).toContain("fix the bug"); + it('solve prompt names the checkout root and the venv python', () => { + const p = buildSwebenchSolvePrompt( + 'fix the bug', + '/workspace/co-run-1', + '/workspace/venv-run-1/bin/python', + ); + expect(p).toContain('/workspace/co-run-1'); + expect(p).toContain('/workspace/venv-run-1/bin/python'); + expect(p).toContain('fix the bug'); }); }); const swebenchArgs = { - repoUrl: "/repos/django/django.git", - baseCommit: "abc1234", - envKey: "sweb.env.py.x86_64.deadbeef:latest", - runId: "run-1", + repoUrl: '/repos/django/django.git', + baseCommit: 'abc1234', + envKey: 'sweb.env.py.x86_64.deadbeef:latest', + runId: 'run-1', }; -describe("setupSwebenchWorkspace", () => { - it("returns trimmed stdout as the checkout dir on success", async () => { +describe('setupSwebenchWorkspace', () => { + it('returns trimmed stdout as the checkout dir on success', async () => { const transport = { - exec: async () => ({ stdout: Buffer.from("/workspace/co-run-1\n"), exitCode: 0, truncated: false }), + exec: async () => ({ + stdout: Buffer.from('/workspace/co-run-1\n'), + exitCode: 0, + truncated: false, + }), close: async () => {}, }; - expect(await setupSwebenchWorkspace(transport, swebenchArgs)).toBe("/workspace/co-run-1"); + expect(await setupSwebenchWorkspace(transport, swebenchArgs)).toBe('/workspace/co-run-1'); }); - it("throws on non-zero exit", async () => { + it('throws on non-zero exit', async () => { const transport = { - exec: async () => ({ stdout: Buffer.from(""), exitCode: 1, truncated: false }), + exec: async () => ({ stdout: Buffer.from(''), exitCode: 1, truncated: false }), close: async () => {}, }; - await expect(setupSwebenchWorkspace(transport, swebenchArgs)).rejects.toThrow(/swebench setup failed/); + await expect(setupSwebenchWorkspace(transport, swebenchArgs)).rejects.toThrow( + /swebench setup failed/, + ); }); - it("reports a capped setup as truncation, not a failed setup", async () => { + it('reports a capped setup as truncation, not a failed setup', async () => { // A setup whose clone/venv/install output overruns the sandbox output cap currently // surfaces as "swebench setup failed (exit null)", which reads as a broken command // rather than output too large for the seam. const transport = { - exec: async () => ({ stdout: Buffer.from("partial"), exitCode: null, truncated: true }), + exec: async () => ({ stdout: Buffer.from('partial'), exitCode: null, truncated: true }), close: async () => {}, }; await expect(setupSwebenchWorkspace(transport, swebenchArgs)).rejects.toThrow(/output cap/); }); }); -describe("captureSwebenchDiff", () => { - it("returns stdout as the patch on exit 0", async () => { +describe('captureSwebenchDiff', () => { + it('returns stdout as the patch on exit 0', async () => { const transport = { - exec: async () => ({ stdout: Buffer.from("diff --git a/x b/x\n"), exitCode: 0, truncated: false }), + exec: async () => ({ + stdout: Buffer.from('diff --git a/x b/x\n'), + exitCode: 0, + truncated: false, + }), close: async () => {}, }; - expect(await captureSwebenchDiff(transport, "run-1")).toBe("diff --git a/x b/x\n"); + expect(await captureSwebenchDiff(transport, 'run-1')).toBe('diff --git a/x b/x\n'); }); - it("throws on non-zero exit", async () => { + it('throws on non-zero exit', async () => { const transport = { - exec: async () => ({ stdout: Buffer.from(""), exitCode: 3, truncated: false }), + exec: async () => ({ stdout: Buffer.from(''), exitCode: 3, truncated: false }), close: async () => {}, }; - await expect(captureSwebenchDiff(transport, "run-1")).rejects.toThrow(/exit 3/); + await expect(captureSwebenchDiff(transport, 'run-1')).rejects.toThrow(/exit 3/); }); - it("reports a capped diff as truncation, not a failed capture", async () => { + it('reports a capped diff as truncation, not a failed capture', async () => { // A >8 MiB diff currently surfaces as "swebench diff capture failed (exit null)", // which reads as a broken git command rather than a diff too large for the seam. const transport = { - exec: async () => ({ stdout: Buffer.from("partial"), exitCode: null, truncated: true }), + exec: async () => ({ stdout: Buffer.from('partial'), exitCode: null, truncated: true }), close: async () => {}, }; - await expect(captureSwebenchDiff(transport, "run-1")).rejects.toThrow(/output cap/); + await expect(captureSwebenchDiff(transport, 'run-1')).rejects.toThrow(/output cap/); }); }); diff --git a/harness/test/tool-choice-extension.test.ts b/harness/test/tool-choice-extension.test.ts index af1e725..8aa448b 100644 --- a/harness/test/tool-choice-extension.test.ts +++ b/harness/test/tool-choice-extension.test.ts @@ -1,12 +1,16 @@ -import { describe, it, expect, beforeEach, afterEach } from "vitest"; -import { toolChoiceExtension } from "../src/tool-choice-extension"; +import { describe, it, expect, beforeEach, afterEach } from 'vitest'; +import { toolChoiceExtension } from '../src/tool-choice-extension'; type Handler = (e: { payload?: unknown }) => unknown; // Minimal fake `pi` that captures the before_provider_request handler the extension registers. function makePi(): { pi: unknown; getHandler: () => Handler | undefined } { let handler: Handler | undefined; - const pi = { on: (_event: string, h: Handler) => { handler = h; } }; + const pi = { + on: (_event: string, h: Handler) => { + handler = h; + }, + }; return { pi, getHandler: () => handler }; } @@ -15,7 +19,7 @@ const invoke = (h: Handler, tools: unknown, toolChoice?: unknown) => tool_choice?: unknown; }; -describe("toolChoiceExtension", () => { +describe('toolChoiceExtension', () => { const saved = { custom: process.env.SH_MODEL_CUSTOM, api: process.env.SH_MODEL_API }; beforeEach(() => { delete process.env.SH_MODEL_CUSTOM; @@ -25,43 +29,45 @@ describe("toolChoiceExtension", () => { saved.custom === undefined ? delete process.env.SH_MODEL_CUSTOM : (process.env.SH_MODEL_CUSTOM = saved.custom); - saved.api === undefined ? delete process.env.SH_MODEL_API : (process.env.SH_MODEL_API = saved.api); + saved.api === undefined + ? delete process.env.SH_MODEL_API + : (process.env.SH_MODEL_API = saved.api); }); - it("is inert (registers no handler) unless SH_MODEL_CUSTOM=1", () => { + it('is inert (registers no handler) unless SH_MODEL_CUSTOM=1', () => { const { pi, getHandler } = makePi(); toolChoiceExtension()(pi as never); expect(getHandler()).toBeUndefined(); }); - it("injects the Anthropic object form by default (SH_MODEL_API unset)", () => { - process.env.SH_MODEL_CUSTOM = "1"; + it('injects the Anthropic object form by default (SH_MODEL_API unset)', () => { + process.env.SH_MODEL_CUSTOM = '1'; const { pi, getHandler } = makePi(); toolChoiceExtension()(pi as never); - const out = invoke(getHandler()!, [{ name: "bash" }]); - expect(out.tool_choice).toEqual({ type: "auto" }); + const out = invoke(getHandler()!, [{ name: 'bash' }]); + expect(out.tool_choice).toEqual({ type: 'auto' }); }); - it("injects the OpenAI string form when SH_MODEL_API=openai-completions", () => { - process.env.SH_MODEL_CUSTOM = "1"; - process.env.SH_MODEL_API = "openai-completions"; + it('injects the OpenAI string form when SH_MODEL_API=openai-completions', () => { + process.env.SH_MODEL_CUSTOM = '1'; + process.env.SH_MODEL_API = 'openai-completions'; const { pi, getHandler } = makePi(); toolChoiceExtension()(pi as never); - const out = invoke(getHandler()!, [{ name: "bash" }]); - expect(out.tool_choice).toBe("auto"); + const out = invoke(getHandler()!, [{ name: 'bash' }]); + expect(out.tool_choice).toBe('auto'); }); - it("does not override an already-set tool_choice", () => { - process.env.SH_MODEL_CUSTOM = "1"; - process.env.SH_MODEL_API = "openai-completions"; + it('does not override an already-set tool_choice', () => { + process.env.SH_MODEL_CUSTOM = '1'; + process.env.SH_MODEL_API = 'openai-completions'; const { pi, getHandler } = makePi(); toolChoiceExtension()(pi as never); - const out = invoke(getHandler()!, [{ name: "bash" }], "required"); - expect(out.tool_choice).toBe("required"); + const out = invoke(getHandler()!, [{ name: 'bash' }], 'required'); + expect(out.tool_choice).toBe('required'); }); - it("leaves tool_choice unset when there are no tools", () => { - process.env.SH_MODEL_CUSTOM = "1"; + it('leaves tool_choice unset when there are no tools', () => { + process.env.SH_MODEL_CUSTOM = '1'; const { pi, getHandler } = makePi(); toolChoiceExtension()(pi as never); const out = invoke(getHandler()!, []); diff --git a/harness/test/turn-stream.test.ts b/harness/test/turn-stream.test.ts index 494cb56..2b94d23 100644 --- a/harness/test/turn-stream.test.ts +++ b/harness/test/turn-stream.test.ts @@ -1,6 +1,12 @@ -import { describe, it, expect, afterEach } from "vitest"; -import { sseExtension, clip, previewCap, terminalFrame, type TurnStreamFrame } from "../src/turn-stream.js"; -import type { TurnResult } from "../src/run-turn.js"; +import { describe, it, expect, afterEach } from 'vitest'; +import { + sseExtension, + clip, + previewCap, + terminalFrame, + type TurnStreamFrame, +} from '../src/turn-stream.js'; +import type { TurnResult } from '../src/run-turn.js'; // A fake Pi that captures the handler registered per event NAME, plus emit() to fire one. // sseExtension registers multiple handlers (message_update / tool_execution_start / @@ -30,124 +36,139 @@ function drive(opts?: { previewBytes?: number }) { return { emit, frames }; } -describe("sseExtension frame translation", () => { - it("translates text_delta / thinking_delta / tool start+end into the frame sequence", () => { +describe('sseExtension frame translation', () => { + it('translates text_delta / thinking_delta / tool start+end into the frame sequence', () => { const { emit, frames } = drive(); - emit("message_update", { assistantMessageEvent: { type: "text_delta", delta: "Hel" } }); - emit("message_update", { assistantMessageEvent: { type: "thinking_delta", delta: "hmm" } }); - emit("tool_execution_start", { toolCallId: "t1", toolName: "bash", args: { cmd: "ls" } }); - emit("tool_execution_end", { toolCallId: "t1", toolName: "bash", result: "file.txt", isError: false }); + emit('message_update', { assistantMessageEvent: { type: 'text_delta', delta: 'Hel' } }); + emit('message_update', { assistantMessageEvent: { type: 'thinking_delta', delta: 'hmm' } }); + emit('tool_execution_start', { toolCallId: 't1', toolName: 'bash', args: { cmd: 'ls' } }); + emit('tool_execution_end', { + toolCallId: 't1', + toolName: 'bash', + result: 'file.txt', + isError: false, + }); expect(frames).toEqual([ - { type: "text", delta: "Hel" }, - { type: "thinking", delta: "hmm" }, - { type: "tool_use", id: "t1", name: "bash", args: { cmd: "ls" } }, - { type: "tool_result", id: "t1", isError: false, preview: "file.txt" }, + { type: 'text', delta: 'Hel' }, + { type: 'thinking', delta: 'hmm' }, + { type: 'tool_use', id: 't1', name: 'bash', args: { cmd: 'ls' } }, + { type: 'tool_result', id: 't1', isError: false, preview: 'file.txt' }, ]); }); it("drops empty deltas (no frame for delta === '')", () => { const { emit, frames } = drive(); - emit("message_update", { assistantMessageEvent: { type: "text_delta", delta: "" } }); - emit("message_update", { assistantMessageEvent: { type: "thinking_delta", delta: "" } }); + emit('message_update', { assistantMessageEvent: { type: 'text_delta', delta: '' } }); + emit('message_update', { assistantMessageEvent: { type: 'thinking_delta', delta: '' } }); expect(frames).toEqual([]); }); - it("propagates isError verbatim on tool_result", () => { + it('propagates isError verbatim on tool_result', () => { const { emit, frames } = drive(); - emit("tool_execution_end", { toolCallId: "t2", toolName: "bash", result: "boom", isError: true }); - expect(frames[0]).toEqual({ type: "tool_result", id: "t2", isError: true, preview: "boom" }); + emit('tool_execution_end', { + toolCallId: 't2', + toolName: 'bash', + result: 'boom', + isError: true, + }); + expect(frames[0]).toEqual({ type: 'tool_result', id: 't2', isError: true, preview: 'boom' }); }); }); -describe("clip / fidelity-B truncation", () => { - it("returns short results unchanged", () => { - expect(clip("hi", 2048)).toBe("hi"); +describe('clip / fidelity-B truncation', () => { + it('returns short results unchanged', () => { + expect(clip('hi', 2048)).toBe('hi'); }); - it("clips an oversized result to the byte cap and marks it truncated", () => { - const big = "x".repeat(5000); + it('clips an oversized result to the byte cap and marks it truncated', () => { + const big = 'x'.repeat(5000); const out = clip(big, 2048); - expect(out.startsWith("x".repeat(2048))).toBe(true); - expect(out.endsWith("…[truncated]")).toBe(true); + expect(out.startsWith('x'.repeat(2048))).toBe(true); + expect(out.endsWith('…[truncated]')).toBe(true); // the un-suffixed head is exactly the cap in bytes - expect(Buffer.from(out.slice(0, -"…[truncated]".length), "utf8").byteLength).toBe(2048); + expect(Buffer.from(out.slice(0, -'…[truncated]'.length), 'utf8').byteLength).toBe(2048); }); - it("coerces non-string results via JSON before clipping", () => { + it('coerces non-string results via JSON before clipping', () => { expect(clip({ a: 1 }, 2048)).toBe('{"a":1}'); }); }); -describe("previewCap resolution (override > env > default)", () => { +describe('previewCap resolution (override > env > default)', () => { const DEFAULT = 2048; - const ENV = "SH_TURN_STREAM_TOOL_RESULT_PREVIEW_BYTES"; + const ENV = 'SH_TURN_STREAM_TOOL_RESULT_PREVIEW_BYTES'; afterEach(() => { delete process.env[ENV]; // isolate the env-branch cases from each other and the default }); - it("uses a finite, non-negative override verbatim (0 is a valid cap)", () => { + it('uses a finite, non-negative override verbatim (0 is a valid cap)', () => { expect(previewCap(512)).toBe(512); expect(previewCap(0)).toBe(0); }); - it("clamps a non-finite or negative override back to the default", () => { + it('clamps a non-finite or negative override back to the default', () => { expect(previewCap(-1)).toBe(DEFAULT); expect(previewCap(Number.NaN)).toBe(DEFAULT); expect(previewCap(Number.POSITIVE_INFINITY)).toBe(DEFAULT); }); - it("falls back to the default when neither override nor env is set", () => { + it('falls back to the default when neither override nor env is set', () => { expect(previewCap()).toBe(DEFAULT); }); - it("parses a valid env override when no explicit override is passed", () => { - process.env[ENV] = "1024"; + it('parses a valid env override when no explicit override is passed', () => { + process.env[ENV] = '1024'; expect(previewCap()).toBe(1024); - process.env[ENV] = "0"; + process.env[ENV] = '0'; expect(previewCap()).toBe(0); }); - it("clamps an unparseable or negative env value back to the default", () => { - process.env[ENV] = "not-a-number"; + it('clamps an unparseable or negative env value back to the default', () => { + process.env[ENV] = 'not-a-number'; expect(previewCap()).toBe(DEFAULT); - process.env[ENV] = "-5"; + process.env[ENV] = '-5'; expect(previewCap()).toBe(DEFAULT); }); - it("an explicit override wins over the env value", () => { - process.env[ENV] = "1024"; + it('an explicit override wins over the env value', () => { + process.env[ENV] = '1024'; expect(previewCap(256)).toBe(256); }); }); -describe("terminalFrame selection & parity", () => { - it("clean stop-reason → done carrying sessionId/stopReason/usage", () => { +describe('terminalFrame selection & parity', () => { + it('clean stop-reason → done carrying sessionId/stopReason/usage', () => { const r: TurnResult = { - sessionId: "s1", - response: "hi", - stopReason: "end_turn", + sessionId: 's1', + response: 'hi', + stopReason: 'end_turn', usage: { input: 1, output: 2, cacheRead: 0, cacheWrite: 0, total: 3 }, }; expect(terminalFrame(r)).toEqual({ - type: "done", - sessionId: "s1", - stopReason: "end_turn", + type: 'done', + sessionId: 's1', + stopReason: 'end_turn', usage: { input: 1, output: 2, cacheRead: 0, cacheWrite: 0, total: 3 }, }); }); - it("error stop-reason → error carrying the same errorMessage a sync caller reads", () => { - const r: TurnResult = { sessionId: "s2", response: "", stopReason: "error", errorMessage: "boom" }; + it('error stop-reason → error carrying the same errorMessage a sync caller reads', () => { + const r: TurnResult = { + sessionId: 's2', + response: '', + stopReason: 'error', + errorMessage: 'boom', + }; expect(terminalFrame(r)).toEqual({ - type: "error", - sessionId: "s2", - stopReason: "error", - errorMessage: "boom", + type: 'error', + sessionId: 's2', + stopReason: 'error', + errorMessage: 'boom', }); }); - it("max_tokens is a clean finish → done", () => { - const r: TurnResult = { sessionId: "s3", response: "partial", stopReason: "max_tokens" }; - expect(terminalFrame(r)).toEqual({ type: "done", sessionId: "s3", stopReason: "max_tokens" }); + it('max_tokens is a clean finish → done', () => { + const r: TurnResult = { sessionId: 's3', response: 'partial', stopReason: 'max_tokens' }; + expect(terminalFrame(r)).toEqual({ type: 'done', sessionId: 's3', stopReason: 'max_tokens' }); }); }); diff --git a/harness/test/verdict-recovery.test.ts b/harness/test/verdict-recovery.test.ts index d16932b..bf905a2 100644 --- a/harness/test/verdict-recovery.test.ts +++ b/harness/test/verdict-recovery.test.ts @@ -1,42 +1,50 @@ -import { describe, it, expect } from "vitest"; -import { verdictFromCustomEntry, toSessionId } from "../src/run-leaf"; +import { describe, it, expect } from 'vitest'; +import { verdictFromCustomEntry, toSessionId } from '../src/run-leaf'; -describe("toSessionId", () => { +describe('toSessionId', () => { it("maps the spec's / id to a valid Pi session id", () => { - expect(toSessionId("run-1106/i1")).toBe("run-1106-i1"); + expect(toSessionId('run-1106/i1')).toBe('run-1106-i1'); }); - it("replaces every invalid char and trims to alphanumeric ends", () => { - expect(toSessionId("/a b/c/")).toBe("a-b-c"); - expect(toSessionId("a.b_c-d")).toBe("a.b_c-d"); // dots, underscores, dashes are allowed + it('replaces every invalid char and trims to alphanumeric ends', () => { + expect(toSessionId('/a b/c/')).toBe('a-b-c'); + expect(toSessionId('a.b_c-d')).toBe('a.b_c-d'); // dots, underscores, dashes are allowed }); - it("is deterministic so a retry maps to the same session", () => { - expect(toSessionId("run/x")).toBe(toSessionId("run/x")); + it('is deterministic so a retry maps to the same session', () => { + expect(toSessionId('run/x')).toBe(toSessionId('run/x')); }); - it("falls back to a non-empty id when nothing valid remains", () => { - expect(toSessionId("///")).toBe("leaf"); + it('falls back to a non-empty id when nothing valid remains', () => { + expect(toSessionId('///')).toBe('leaf'); }); }); -const v = { item_id: "i1", verdict: "FLAGGED", reason: "r" }; +const v = { item_id: 'i1', verdict: 'FLAGGED', reason: 'r' }; -describe("verdictFromCustomEntry", () => { - it("recovers a verdict from a verdict custom entry", () => { - expect(verdictFromCustomEntry({ type: "custom", customType: "verdict", data: v })).toEqual(v); +describe('verdictFromCustomEntry', () => { + it('recovers a verdict from a verdict custom entry', () => { + expect(verdictFromCustomEntry({ type: 'custom', customType: 'verdict', data: v })).toEqual(v); }); - it("returns null for a non-verdict custom entry (e.g. a checkpoint marker)", () => { + it('returns null for a non-verdict custom entry (e.g. a checkpoint marker)', () => { expect( - verdictFromCustomEntry({ type: "custom", customType: "checkpoint", data: { resumeFromPosition: 3 } }), + verdictFromCustomEntry({ + type: 'custom', + customType: 'checkpoint', + data: { resumeFromPosition: 3 }, + }), ).toBeNull(); }); - it("returns null for a non-custom entry", () => { - expect(verdictFromCustomEntry({ type: "message", role: "user", content: "hi" })).toBeNull(); + it('returns null for a non-custom entry', () => { + expect(verdictFromCustomEntry({ type: 'message', role: 'user', content: 'hi' })).toBeNull(); }); - it("returns null when the entry data is not a schema-valid verdict", () => { + it('returns null when the entry data is not a schema-valid verdict', () => { expect( - verdictFromCustomEntry({ type: "custom", customType: "verdict", data: { item_id: "i1", verdict: "MAYBE", reason: "r" } }), + verdictFromCustomEntry({ + type: 'custom', + customType: 'verdict', + data: { item_id: 'i1', verdict: 'MAYBE', reason: 'r' }, + }), ).toBeNull(); expect(verdictFromCustomEntry(null)).toBeNull(); expect(verdictFromCustomEntry(undefined)).toBeNull(); diff --git a/harness/test/verdict-termination-extension.test.ts b/harness/test/verdict-termination-extension.test.ts index 55073a4..10728e8 100644 --- a/harness/test/verdict-termination-extension.test.ts +++ b/harness/test/verdict-termination-extension.test.ts @@ -1,60 +1,64 @@ -import { describe, it, expect } from "vitest"; -import { verdictTerminationExtension } from "../src/verdict-termination-extension"; -import type { VerdictCapture } from "../src/submit-verdict-tool"; +import { describe, it, expect } from 'vitest'; +import { verdictTerminationExtension } from '../src/verdict-termination-extension'; +import type { VerdictCapture } from '../src/submit-verdict-tool'; function fakePi() { const handlers: Record = {}; - const pi = { on: (ev: string, h: Function) => { handlers[ev] = h; } }; + const pi = { + on: (ev: string, h: Function) => { + handlers[ev] = h; + }, + }; return { pi: pi as any, handlers }; } -describe("verdictTerminationExtension", () => { - it("does not block tool calls before a verdict is captured and under the turn cap", () => { +describe('verdictTerminationExtension', () => { + it('does not block tool calls before a verdict is captured and under the turn cap', () => { const capture: VerdictCapture = {}; const { pi, handlers } = fakePi(); verdictTerminationExtension(capture, { maxTurns: 5 })(pi); - const result = handlers["tool_call"]({}); + const result = handlers['tool_call']({}); expect(result).toEqual({}); }); - it("blocks tool calls once a verdict has already been captured", () => { - const capture: VerdictCapture = { verdict: { item_id: "i1", verdict: "CLEAR", reason: "r" } }; + it('blocks tool calls once a verdict has already been captured', () => { + const capture: VerdictCapture = { verdict: { item_id: 'i1', verdict: 'CLEAR', reason: 'r' } }; const { pi, handlers } = fakePi(); verdictTerminationExtension(capture, { maxTurns: 5 })(pi); - const result = handlers["tool_call"]({}); + const result = handlers['tool_call']({}); expect(result.block).toBe(true); expect(result.reason).toMatch(/verdict already submitted/i); }); - it("blocks tool calls once maxTurns is exceeded without a verdict", () => { + it('blocks tool calls once maxTurns is exceeded without a verdict', () => { const capture: VerdictCapture = {}; const { pi, handlers } = fakePi(); verdictTerminationExtension(capture, { maxTurns: 2 })(pi); // turn_start fires once per turn; simulate 3 turns (exceeds cap of 2) - handlers["turn_start"](); - handlers["turn_start"](); - handlers["turn_start"](); - const result = handlers["tool_call"]({}); + handlers['turn_start'](); + handlers['turn_start'](); + handlers['turn_start'](); + const result = handlers['tool_call']({}); expect(result.block).toBe(true); expect(result.reason).toMatch(/turn limit/i); }); - it("does not block at exactly the turn cap (only after exceeding it)", () => { + it('does not block at exactly the turn cap (only after exceeding it)', () => { const capture: VerdictCapture = {}; const { pi, handlers } = fakePi(); verdictTerminationExtension(capture, { maxTurns: 2 })(pi); - handlers["turn_start"](); - handlers["turn_start"](); - const result = handlers["tool_call"]({}); + handlers['turn_start'](); + handlers['turn_start'](); + const result = handlers['tool_call']({}); expect(result).toEqual({}); }); - it("applies a default turn cap when maxTurns is omitted (never leaves the loop unbounded)", () => { + it('applies a default turn cap when maxTurns is omitted (never leaves the loop unbounded)', () => { const capture: VerdictCapture = {}; const { pi, handlers } = fakePi(); verdictTerminationExtension(capture)(pi); - for (let i = 0; i < 41; i++) handlers["turn_start"](); - const result = handlers["tool_call"]({}); + for (let i = 0; i < 41; i++) handlers['turn_start'](); + const result = handlers['tool_call']({}); expect(result.block).toBe(true); expect(result.reason).toMatch(/turn limit/i); }); @@ -64,8 +68,8 @@ describe("verdictTerminationExtension", () => { const { pi, handlers } = fakePi(); verdictTerminationExtension(capture, { maxTurns: 0 })(pi); // 0 must NOT mean unbounded — it should fall back to the default cap and still block. - for (let i = 0; i < 41; i++) handlers["turn_start"](); - const result = handlers["tool_call"]({}); + for (let i = 0; i < 41; i++) handlers['turn_start'](); + const result = handlers['tool_call']({}); expect(result.block).toBe(true); expect(result.reason).toMatch(/turn limit/i); }); diff --git a/harness/test/verdict.test.ts b/harness/test/verdict.test.ts index 35c0438..f89f2dc 100644 --- a/harness/test/verdict.test.ts +++ b/harness/test/verdict.test.ts @@ -1,24 +1,27 @@ -import { describe, it, expect } from "vitest"; -import { validateVerdict } from "../src/verdict"; +import { describe, it, expect } from 'vitest'; +import { validateVerdict } from '../src/verdict'; -describe("validateVerdict", () => { - it("accepts a well-formed verdict", () => { - const r = validateVerdict({ item_id: "i1", verdict: "FLAGGED", reason: "calls eval on input" }); - expect(r).toEqual({ ok: true, value: { item_id: "i1", verdict: "FLAGGED", reason: "calls eval on input" } }); +describe('validateVerdict', () => { + it('accepts a well-formed verdict', () => { + const r = validateVerdict({ item_id: 'i1', verdict: 'FLAGGED', reason: 'calls eval on input' }); + expect(r).toEqual({ + ok: true, + value: { item_id: 'i1', verdict: 'FLAGGED', reason: 'calls eval on input' }, + }); }); - it("rejects an unknown verdict label", () => { - const r = validateVerdict({ item_id: "i1", verdict: "MAYBE", reason: "x" }); + it('rejects an unknown verdict label', () => { + const r = validateVerdict({ item_id: 'i1', verdict: 'MAYBE', reason: 'x' }); expect(r.ok).toBe(false); }); - it("rejects missing fields", () => { - expect(validateVerdict({ item_id: "i1", verdict: "CLEAR" }).ok).toBe(false); - expect(validateVerdict({ verdict: "CLEAR", reason: "x" }).ok).toBe(false); + it('rejects missing fields', () => { + expect(validateVerdict({ item_id: 'i1', verdict: 'CLEAR' }).ok).toBe(false); + expect(validateVerdict({ verdict: 'CLEAR', reason: 'x' }).ok).toBe(false); }); - it("rejects non-object input", () => { + it('rejects non-object input', () => { expect(validateVerdict(null).ok).toBe(false); - expect(validateVerdict("nope").ok).toBe(false); + expect(validateVerdict('nope').ok).toBe(false); }); }); diff --git a/harness/vitest.config.ts b/harness/vitest.config.ts index c794660..e1e28f8 100644 --- a/harness/vitest.config.ts +++ b/harness/vitest.config.ts @@ -1,5 +1,5 @@ -import { defineConfig } from "vitest/config"; +import { defineConfig } from 'vitest/config'; export default defineConfig({ - test: { include: ["test/**/*.test.ts"] }, + test: { include: ['test/**/*.test.ts'] }, }); diff --git a/packages/ibac-stub/src/decide.ts b/packages/ibac-stub/src/decide.ts index 55bb165..e11b6cc 100644 --- a/packages/ibac-stub/src/decide.ts +++ b/packages/ibac-stub/src/decide.ts @@ -9,14 +9,18 @@ export type DenyRules = { denyArgMarkers?: string[]; }; -export type Verdict = { verdict: "allow" | "deny"; reason: string }; +export type Verdict = { verdict: 'allow' | 'deny'; reason: string }; -type RuleKind = "tool" | "url" | "arg marker"; +type RuleKind = 'tool' | 'url' | 'arg marker'; -function findMatch(actionText: string, kind: RuleKind, entries: string[] | undefined): Verdict | undefined { +function findMatch( + actionText: string, + kind: RuleKind, + entries: string[] | undefined, +): Verdict | undefined { for (const entry of entries ?? []) { - if (entry !== "" && actionText.includes(entry)) { - return { verdict: "deny", reason: `denied: ${kind} '${entry}'` }; + if (entry !== '' && actionText.includes(entry)) { + return { verdict: 'deny', reason: `denied: ${kind} '${entry}'` }; } } return undefined; @@ -24,11 +28,11 @@ function findMatch(actionText: string, kind: RuleKind, entries: string[] | undef export function decide(actionText: string, rules: DenyRules): Verdict { return ( - findMatch(actionText, "tool", rules.denyTools) ?? - findMatch(actionText, "url", rules.denyUrlSubstrings) ?? - findMatch(actionText, "arg marker", rules.denyArgMarkers) ?? { - verdict: "allow", - reason: "no matching deny rule", + findMatch(actionText, 'tool', rules.denyTools) ?? + findMatch(actionText, 'url', rules.denyUrlSubstrings) ?? + findMatch(actionText, 'arg marker', rules.denyArgMarkers) ?? { + verdict: 'allow', + reason: 'no matching deny rule', } ); } diff --git a/packages/ibac-stub/src/index.ts b/packages/ibac-stub/src/index.ts index 5fa6714..83f460d 100644 --- a/packages/ibac-stub/src/index.ts +++ b/packages/ibac-stub/src/index.ts @@ -1,3 +1,3 @@ -export { decide, type DenyRules, type Verdict } from "./decide.js"; -export { startServer, buildHandler, extractActionText } from "./server.js"; -export { rulesFromEnv, portFromEnv } from "./main.js"; +export { decide, type DenyRules, type Verdict } from './decide.js'; +export { startServer, buildHandler, extractActionText } from './server.js'; +export { rulesFromEnv, portFromEnv } from './main.js'; diff --git a/packages/ibac-stub/src/main.ts b/packages/ibac-stub/src/main.ts index 648e4fd..f4272c5 100644 --- a/packages/ibac-stub/src/main.ts +++ b/packages/ibac-stub/src/main.ts @@ -1,6 +1,6 @@ -import { fileURLToPath } from "node:url"; -import { startServer } from "./server.js"; -import type { DenyRules } from "./decide.js"; +import { fileURLToPath } from 'node:url'; +import { startServer } from './server.js'; +import type { DenyRules } from './decide.js'; // Comma-separated env var → trimmed, non-empty entries. `undefined` (unset) yields `undefined` so // decide()'s `?? []` default applies, rather than an explicit empty array either way — the @@ -10,7 +10,7 @@ function listFromEnv(name: string, env: NodeJS.ProcessEnv = process.env): string const raw = env[name]; if (raw === undefined) return undefined; const entries = raw - .split(",") + .split(',') .map((s) => s.trim()) .filter((s) => s.length > 0); return entries.length > 0 ? entries : undefined; @@ -18,9 +18,9 @@ function listFromEnv(name: string, env: NodeJS.ProcessEnv = process.env): string export function rulesFromEnv(env: NodeJS.ProcessEnv = process.env): DenyRules { return { - denyTools: listFromEnv("IBAC_STUB_DENY_TOOLS", env), - denyUrlSubstrings: listFromEnv("IBAC_STUB_DENY_URLS", env), - denyArgMarkers: listFromEnv("IBAC_STUB_DENY_ARG_MARKERS", env), + denyTools: listFromEnv('IBAC_STUB_DENY_TOOLS', env), + denyUrlSubstrings: listFromEnv('IBAC_STUB_DENY_URLS', env), + denyArgMarkers: listFromEnv('IBAC_STUB_DENY_ARG_MARKERS', env), }; } diff --git a/packages/ibac-stub/src/server.ts b/packages/ibac-stub/src/server.ts index 086aa5c..a863be8 100644 --- a/packages/ibac-stub/src/server.ts +++ b/packages/ibac-stub/src/server.ts @@ -1,7 +1,7 @@ -import { createServer, type IncomingMessage, type ServerResponse, type Server } from "node:http"; -import { decide, type DenyRules } from "./decide.js"; +import { createServer, type IncomingMessage, type ServerResponse, type Server } from 'node:http'; +import { decide, type DenyRules } from './decide.js'; -const JSON_HEADERS = { "Content-Type": "application/json" }; +const JSON_HEADERS = { 'Content-Type': 'application/json' }; type ChatMessage = { role?: string; content?: unknown }; type ChatCompletionRequest = { model?: string; messages?: ChatMessage[] }; @@ -9,9 +9,9 @@ type ChatCompletionRequest = { model?: string; messages?: ChatMessage[] }; function readBody(req: IncomingMessage): Promise { return new Promise((resolve, reject) => { const chunks: Buffer[] = []; - req.on("data", (chunk: Buffer) => chunks.push(chunk)); - req.on("end", () => resolve(Buffer.concat(chunks).toString())); - req.on("error", reject); + req.on('data', (chunk: Buffer) => chunks.push(chunk)); + req.on('end', () => resolve(Buffer.concat(chunks).toString())); + req.on('error', reject); }); } @@ -19,35 +19,39 @@ function readBody(req: IncomingMessage): Promise { // args) in the text of the request's `user` messages. Concatenate them to get the text decide() // matches against. export function extractActionText(body: ChatCompletionRequest): string { - if (typeof body !== "object" || body === null) return ""; + if (typeof body !== 'object' || body === null) return ''; return (body.messages ?? []) - .filter((m) => m.role === "user") - .map((m) => (typeof m.content === "string" ? m.content : "")) - .join("\n"); + .filter((m) => m.role === 'user') + .map((m) => (typeof m.content === 'string' ? m.content : '')) + .join('\n'); } function chatCompletionEnvelope(model: string | undefined, content: string) { return { - id: "ibac-stub-0", - object: "chat.completion", + id: 'ibac-stub-0', + object: 'chat.completion', created: Math.floor(Date.now() / 1000), - model: model ?? "ibac-stub", + model: model ?? 'ibac-stub', choices: [ { index: 0, - message: { role: "assistant", content }, - finish_reason: "stop", + message: { role: 'assistant', content }, + finish_reason: 'stop', }, ], }; } -async function handleChatCompletions(req: IncomingMessage, res: ServerResponse, rules: DenyRules): Promise { +async function handleChatCompletions( + req: IncomingMessage, + res: ServerResponse, + rules: DenyRules, +): Promise { let raw: string; try { raw = await readBody(req); } catch { - res.writeHead(400, JSON_HEADERS).end(JSON.stringify({ error: "read_error" })); + res.writeHead(400, JSON_HEADERS).end(JSON.stringify({ error: 'read_error' })); return; } @@ -55,26 +59,31 @@ async function handleChatCompletions(req: IncomingMessage, res: ServerResponse, try { parsed = JSON.parse(raw); } catch { - res.writeHead(400, JSON_HEADERS).end(JSON.stringify({ error: "invalid_json" })); + res.writeHead(400, JSON_HEADERS).end(JSON.stringify({ error: 'invalid_json' })); return; } const actionText = extractActionText(parsed); const result = decide(actionText, rules); const content = JSON.stringify(result); - res.writeHead(200, JSON_HEADERS).end(JSON.stringify(chatCompletionEnvelope(parsed?.model, content))); + res + .writeHead(200, JSON_HEADERS) + .end(JSON.stringify(chatCompletionEnvelope(parsed?.model, content))); } -export function buildHandler(rules: DenyRules): (req: IncomingMessage, res: ServerResponse) => void { +export function buildHandler( + rules: DenyRules, +): (req: IncomingMessage, res: ServerResponse) => void { return (req, res) => { - if (req.method === "GET" && req.url === "/healthz") { - res.writeHead(200).end("ok"); + if (req.method === 'GET' && req.url === '/healthz') { + res.writeHead(200).end('ok'); return; } - if (req.method === "POST" && req.url === "/v1/chat/completions") { + if (req.method === 'POST' && req.url === '/v1/chat/completions') { handleChatCompletions(req, res, rules).catch((err) => { - if (!res.headersSent) res.writeHead(500, JSON_HEADERS).end(JSON.stringify({ error: String(err) })); + if (!res.headersSent) + res.writeHead(500, JSON_HEADERS).end(JSON.stringify({ error: String(err) })); }); return; } @@ -86,7 +95,7 @@ export function buildHandler(rules: DenyRules): (req: IncomingMessage, res: Serv export function startServer(rules: DenyRules, port = 8080): Server { const server = createServer(buildHandler(rules)); - process.on("SIGTERM", () => { + process.on('SIGTERM', () => { server.close(() => process.exit(0)); }); diff --git a/packages/ibac-stub/test/decide.test.ts b/packages/ibac-stub/test/decide.test.ts index c53ff28..a627660 100644 --- a/packages/ibac-stub/test/decide.test.ts +++ b/packages/ibac-stub/test/decide.test.ts @@ -1,43 +1,43 @@ -import { describe, it, expect } from "vitest"; -import { decide } from "../src/decide.js"; +import { describe, it, expect } from 'vitest'; +import { decide } from '../src/decide.js'; -describe("decide", () => { - it("allows by default (empty rules)", () => { - const result = decide("GET https://example.com/anything", {}); - expect(result.verdict).toBe("allow"); +describe('decide', () => { + it('allows by default (empty rules)', () => { + const result = decide('GET https://example.com/anything', {}); + expect(result.verdict).toBe('allow'); }); - it("denies when action text contains a denyTools entry", () => { - const result = decide("tool call: delete_repo({\"repo\":\"foo\"})", { - denyTools: ["delete_repo"], + it('denies when action text contains a denyTools entry', () => { + const result = decide('tool call: delete_repo({"repo":"foo"})', { + denyTools: ['delete_repo'], }); - expect(result.verdict).toBe("deny"); + expect(result.verdict).toBe('deny'); }); - it("denies when action text contains a denyUrlSubstrings entry", () => { - const result = decide("POST https://internal.example.com/admin/wipe", { - denyUrlSubstrings: ["internal.example.com/admin"], + it('denies when action text contains a denyUrlSubstrings entry', () => { + const result = decide('POST https://internal.example.com/admin/wipe', { + denyUrlSubstrings: ['internal.example.com/admin'], }); - expect(result.verdict).toBe("deny"); + expect(result.verdict).toBe('deny'); }); - it("denies when action text contains a denyArgMarkers entry", () => { - const result = decide("tool call: run_cmd({\"cmd\":\"rm -rf /\"})", { - denyArgMarkers: ["rm -rf /"], + it('denies when action text contains a denyArgMarkers entry', () => { + const result = decide('tool call: run_cmd({"cmd":"rm -rf /"})', { + denyArgMarkers: ['rm -rf /'], }); - expect(result.verdict).toBe("deny"); + expect(result.verdict).toBe('deny'); }); - it("reason is non-empty and names the rule on deny", () => { - const result = decide("tool call: delete_repo({})", { - denyTools: ["delete_repo"], + it('reason is non-empty and names the rule on deny', () => { + const result = decide('tool call: delete_repo({})', { + denyTools: ['delete_repo'], }); expect(result.reason.length).toBeGreaterThan(0); - expect(result.reason).toContain("delete_repo"); + expect(result.reason).toContain('delete_repo'); }); - it("allow reason is set", () => { - const result = decide("GET https://example.com/anything", {}); + it('allow reason is set', () => { + const result = decide('GET https://example.com/anything', {}); expect(result.reason.length).toBeGreaterThan(0); }); }); diff --git a/packages/ibac-stub/test/main.test.ts b/packages/ibac-stub/test/main.test.ts index 3d49c9f..b8ec57e 100644 --- a/packages/ibac-stub/test/main.test.ts +++ b/packages/ibac-stub/test/main.test.ts @@ -1,22 +1,22 @@ -import { describe, it, expect } from "vitest"; -import { rulesFromEnv } from "../src/main.js"; +import { describe, it, expect } from 'vitest'; +import { rulesFromEnv } from '../src/main.js'; -describe("rulesFromEnv", () => { - it("reads deny lists from a caller-supplied env object, not process.env", () => { +describe('rulesFromEnv', () => { + it('reads deny lists from a caller-supplied env object, not process.env', () => { const env = { - IBAC_STUB_DENY_TOOLS: "delete_repo, wipe_db", - IBAC_STUB_DENY_URLS: "internal.example.com/admin", - IBAC_STUB_DENY_ARG_MARKERS: "rm -rf /", + IBAC_STUB_DENY_TOOLS: 'delete_repo, wipe_db', + IBAC_STUB_DENY_URLS: 'internal.example.com/admin', + IBAC_STUB_DENY_ARG_MARKERS: 'rm -rf /', }; const rules = rulesFromEnv(env); - expect(rules.denyTools).toEqual(["delete_repo", "wipe_db"]); - expect(rules.denyUrlSubstrings).toEqual(["internal.example.com/admin"]); - expect(rules.denyArgMarkers).toEqual(["rm -rf /"]); + expect(rules.denyTools).toEqual(['delete_repo', 'wipe_db']); + expect(rules.denyUrlSubstrings).toEqual(['internal.example.com/admin']); + expect(rules.denyArgMarkers).toEqual(['rm -rf /']); }); - it("returns undefined entries when the supplied env has no matching vars set", () => { + it('returns undefined entries when the supplied env has no matching vars set', () => { const rules = rulesFromEnv({}); expect(rules.denyTools).toBeUndefined(); diff --git a/packages/ibac-stub/test/server.test.ts b/packages/ibac-stub/test/server.test.ts index 7ed8a10..909480a 100644 --- a/packages/ibac-stub/test/server.test.ts +++ b/packages/ibac-stub/test/server.test.ts @@ -1,15 +1,15 @@ -import { describe, it, expect, beforeAll, afterAll } from "vitest"; -import type { Server } from "node:http"; -import { startServer } from "../src/server.js"; +import { describe, it, expect, beforeAll, afterAll } from 'vitest'; +import type { Server } from 'node:http'; +import { startServer } from '../src/server.js'; let server: Server; let baseUrl: string; beforeAll(async () => { - server = startServer({ denyTools: ["delete_repo"] }, 0); // port 0 = random available port - await new Promise((resolve) => server.on("listening", resolve)); + server = startServer({ denyTools: ['delete_repo'] }, 0); // port 0 = random available port + await new Promise((resolve) => server.on('listening', resolve)); const addr = server.address(); - if (addr && typeof addr === "object") { + if (addr && typeof addr === 'object') { baseUrl = `http://127.0.0.1:${addr.port}`; } }); @@ -18,69 +18,69 @@ afterAll(async () => { await new Promise((resolve) => server.close(() => resolve())); }); -describe("GET /healthz", () => { - it("returns 200", async () => { +describe('GET /healthz', () => { + it('returns 200', async () => { const res = await fetch(`${baseUrl}/healthz`); expect(res.status).toBe(200); }); }); -describe("POST /v1/chat/completions", () => { - it("returns an allow verdict for a benign action", async () => { +describe('POST /v1/chat/completions', () => { + it('returns an allow verdict for a benign action', async () => { const res = await fetch(`${baseUrl}/v1/chat/completions`, { - method: "POST", - headers: { "content-type": "application/json" }, + method: 'POST', + headers: { 'content-type': 'application/json' }, body: JSON.stringify({ - model: "ibac-judge", - messages: [{ role: "user", content: "GET https://example.com/status" }], + model: 'ibac-judge', + messages: [{ role: 'user', content: 'GET https://example.com/status' }], }), }); expect(res.status).toBe(200); const body = await res.json(); const verdict = JSON.parse(body.choices[0].message.content); - expect(verdict).toEqual({ verdict: "allow", reason: "no matching deny rule" }); + expect(verdict).toEqual({ verdict: 'allow', reason: 'no matching deny rule' }); }); - it("returns a deny verdict when the action text matches a configured deny rule", async () => { + it('returns a deny verdict when the action text matches a configured deny rule', async () => { const res = await fetch(`${baseUrl}/v1/chat/completions`, { - method: "POST", - headers: { "content-type": "application/json" }, + method: 'POST', + headers: { 'content-type': 'application/json' }, body: JSON.stringify({ - model: "ibac-judge", - messages: [{ role: "user", content: "tool call: delete_repo({\"repo\":\"foo\"})" }], + model: 'ibac-judge', + messages: [{ role: 'user', content: 'tool call: delete_repo({"repo":"foo"})' }], }), }); expect(res.status).toBe(200); const body = await res.json(); const verdict = JSON.parse(body.choices[0].message.content); - expect(verdict.verdict).toBe("deny"); - expect(verdict.reason).toContain("delete_repo"); + expect(verdict.verdict).toBe('deny'); + expect(verdict.reason).toContain('delete_repo'); }); - it("returns 400 on invalid JSON", async () => { + it('returns 400 on invalid JSON', async () => { const res = await fetch(`${baseUrl}/v1/chat/completions`, { - method: "POST", - headers: { "content-type": "application/json" }, - body: "not valid json{{{", + method: 'POST', + headers: { 'content-type': 'application/json' }, + body: 'not valid json{{{', }); expect(res.status).toBe(400); }); - it("returns 200 with an allow verdict for a JSON null body", async () => { + it('returns 200 with an allow verdict for a JSON null body', async () => { const res = await fetch(`${baseUrl}/v1/chat/completions`, { - method: "POST", - headers: { "content-type": "application/json" }, - body: "null", + method: 'POST', + headers: { 'content-type': 'application/json' }, + body: 'null', }); expect(res.status).toBe(200); const body = await res.json(); const verdict = JSON.parse(body.choices[0].message.content); - expect(verdict).toEqual({ verdict: "allow", reason: "no matching deny rule" }); + expect(verdict).toEqual({ verdict: 'allow', reason: 'no matching deny rule' }); }); }); -describe("unknown routes", () => { - it("returns 404", async () => { +describe('unknown routes', () => { + it('returns 404', async () => { const res = await fetch(`${baseUrl}/unknown`); expect(res.status).toBe(404); }); diff --git a/packages/ibac-stub/vitest.config.ts b/packages/ibac-stub/vitest.config.ts index c794660..e1e28f8 100644 --- a/packages/ibac-stub/vitest.config.ts +++ b/packages/ibac-stub/vitest.config.ts @@ -1,5 +1,5 @@ -import { defineConfig } from "vitest/config"; +import { defineConfig } from 'vitest/config'; export default defineConfig({ - test: { include: ["test/**/*.test.ts"] }, + test: { include: ['test/**/*.test.ts'] }, }); diff --git a/packages/k8s-sandbox/NOTES-pi-operations.md b/packages/k8s-sandbox/NOTES-pi-operations.md index 4f5fda4..3916d43 100644 --- a/packages/k8s-sandbox/NOTES-pi-operations.md +++ b/packages/k8s-sandbox/NOTES-pi-operations.md @@ -3,6 +3,7 @@ Confirms the M2 design's §4.1 load-bearing risk. ## find — operations SUFFICE + `createFindToolDefinition.execute` (src/core/tools/find.ts:155): when `options.operations.glob` is provided it is used INSTEAD of `fd`. Supplying a custom `glob` that shells out in-pod fully routes find's search to the pod. @@ -11,14 +12,16 @@ Return: array of paths (relative to the search cwd is accepted). (Plan claimed line 154; actual check is at line 155. Behavior matches exactly.) ## grep — operations DO NOT suffice + `createGrepToolDefinition.execute` (src/core/tools/grep.ts): always `spawn(rgPath, args)` against the LOCAL filesystem (line 221). `operations` only feeds `isDirectory` (path check) and `readFile` (context lines). So grep's search runs on the head regardless of operations. Decision: route grep by OVERRIDING the tool's `execute` to run `rg` IN the pod via execInPod, and return grep's result shape: - - No-match: `{ content: [{ type: "text", text: "No matches found" }], details: undefined }` (grep.ts:311) - - Success: `{ content: [{ type: "text", text: output }], details: }` (grep.ts:358) + +- No-match: `{ content: [{ type: "text", text: "No matches found" }], details: undefined }` (grep.ts:311) +- Success: `{ content: [{ type: "text", text: output }], details: }` (grep.ts:358) (Plan cited grep.ts:311 for the success/no-match shape. Actual: line 311 is the no-match resolve; the success resolve is at line 358. Both shapes confirmed above.) diff --git a/packages/k8s-sandbox/README.md b/packages/k8s-sandbox/README.md index b8bb89e..16c93e0 100644 --- a/packages/k8s-sandbox/README.md +++ b/packages/k8s-sandbox/README.md @@ -10,12 +10,12 @@ the serverless harness (Milestone 2). See `harness/cli.ts` registers `k8sSandboxExtension()`. It is **inert unless `KAGENTI_SANDBOX_POD` is set**: -| Env var | Default | Meaning | -|---------|---------|---------| -| `KAGENTI_SANDBOX_POD` | (unset → off) | pod to exec into | -| `KAGENTI_SANDBOX_NAMESPACE` | `default` | namespace | -| `KAGENTI_SANDBOX_CONTEXT` | current-context | kube context | -| `KAGENTI_SANDBOX_CWD` | `/workspace` | pod working dir (announced to the model) | +| Env var | Default | Meaning | +| --------------------------- | --------------- | ---------------------------------------- | +| `KAGENTI_SANDBOX_POD` | (unset → off) | pod to exec into | +| `KAGENTI_SANDBOX_NAMESPACE` | `default` | namespace | +| `KAGENTI_SANDBOX_CONTEXT` | current-context | kube context | +| `KAGENTI_SANDBOX_CWD` | `/workspace` | pod working dir (announced to the model) | Apply the fixture pod first: `kubectl apply -f deploy/sandbox.yaml`. See `SMOKE.md` for the end-to-end runbook. diff --git a/packages/k8s-sandbox/SMOKE.md b/packages/k8s-sandbox/SMOKE.md index 1fdcde4..b6b9856 100644 --- a/packages/k8s-sandbox/SMOKE.md +++ b/packages/k8s-sandbox/SMOKE.md @@ -61,8 +61,8 @@ set, existing pi-fork dist @ submodule `7acc67a`. - **Bare transport baseline:** `kubectl exec -i $POD -- bash -c 'hostname'` → `sandbox-77f89448f6-h8249`. - **Agent turn:** EXIT 0. Assistant reply: - *"Created `proof.txt` in the current directory. The hostname is - **sandbox-77f89448f6-h8249**."* `SESSION_ID=019ed6b2-8d97-7e78-9489-4792b4fe720d`. + _"Created `proof.txt` in the current directory. The hostname is + **sandbox-77f89448f6-h8249**."_ `SESSION_ID=019ed6b2-8d97-7e78-9489-4792b4fe720d`. - **(a) file written in the pod:** `kubectl exec $POD -- cat /workspace/proof.txt` → `sandbox-77f89448f6-h8249` (the pod's hostname). - **(b) head clean:** `HEAD_CLEAN` for `harness/proof.txt`, repo-root `proof.txt`, @@ -133,14 +133,14 @@ Suite is skipped without the gate (`pnpm test` → 47 passed | 6 skipped). - **Claim 3 — env injection:** bash op `echo MARKER=$M3_SMOKE` with `env={M3_SMOKE:"works-42"}` → captured **`MARKER=works-42`**, exit 0. - **Claim 4 — find ignore-list + gitignored directories:** `glob('*.ts', - ignore=['**/node_modules/**','**/.git/**'])` → **`["top.ts","src/keep.ts"]`**. +ignore=['**/node_modules/**','**/.git/**'])` → **`["top.ts","src/keep.ts"]`**. Excluded: `node_modules/pkg/skip.ts`, `.git/cfg.ts` (ignore list), and - **`dist/bundle.ts`** — the gitignored *directory* `dist/` is pruned by rg even + **`dist/bundle.ts`** — the gitignored _directory_ `dist/` is pruned by rg even though `-g '*.ts'` matches inside it (`.gitignore` honoured for dirs). - **Claim 4b — file-level gitignore nuance (verified):** in an isolated `/workspace/ovr` with `.gitignore` = `a.ts`, `glob('*.ts', ignore=[])` → - **`["a.ts","keep2.ts"]`**. The individually-gitignored *file* `a.ts` **is - re-included** — a positive `-g ` whitelist-overrides a *file-level* + **`["a.ts","keep2.ts"]`**. The individually-gitignored _file_ `a.ts` **is + re-included** — a positive `-g ` whitelist-overrides a _file-level_ ignore (minor divergence from Pi's `fd`; see design D5). Net: directory-level gitignore parity; only individually-ignored files matching the glob can leak. - **Claim 5 — close:** `fastExec.close()` non-throwing; the persistent diff --git a/packages/k8s-sandbox/deploy/sandbox.yaml b/packages/k8s-sandbox/deploy/sandbox.yaml index 887556b..6865c38 100644 --- a/packages/k8s-sandbox/deploy/sandbox.yaml +++ b/packages/k8s-sandbox/deploy/sandbox.yaml @@ -4,7 +4,7 @@ metadata: name: sandbox-workspace namespace: default spec: - accessModes: ["ReadWriteOnce"] + accessModes: ['ReadWriteOnce'] resources: requests: storage: 1Gi @@ -30,7 +30,7 @@ spec: - name: sandbox image: alpine:3.20 # Install the tools the Operations rely on, then idle. - command: ["/bin/sh", "-c"] + command: ['/bin/sh', '-c'] args: - apk add --no-cache bash coreutils findutils ripgrep file >/dev/null 2>&1; mkdir -p /workspace; diff --git a/packages/k8s-sandbox/src/config.ts b/packages/k8s-sandbox/src/config.ts index cf0a8f0..055b40e 100644 --- a/packages/k8s-sandbox/src/config.ts +++ b/packages/k8s-sandbox/src/config.ts @@ -21,9 +21,9 @@ export function resolveConfig(env: NodeJS.ProcessEnv, headCwd: string): K8sSandb if (!pod) return null; return { pod, - namespace: env.KAGENTI_SANDBOX_NAMESPACE ?? "default", + namespace: env.KAGENTI_SANDBOX_NAMESPACE ?? 'default', context: env.KAGENTI_SANDBOX_CONTEXT || undefined, - podCwd: env.KAGENTI_SANDBOX_CWD ?? "/workspace", + podCwd: env.KAGENTI_SANDBOX_CWD ?? '/workspace', headCwd, }; } diff --git a/packages/k8s-sandbox/src/exec.ts b/packages/k8s-sandbox/src/exec.ts index 26f2a2f..22a80cd 100644 --- a/packages/k8s-sandbox/src/exec.ts +++ b/packages/k8s-sandbox/src/exec.ts @@ -1,29 +1,29 @@ -import { spawn as nodeSpawn } from "node:child_process"; -import type { K8sSandboxConfig } from "./config.js"; -import type { ExecInPod, SandboxTransport } from "./transport.js"; -import { DEFAULT_OUTPUT_CAP, OUTPUT_TRUNCATED_MARKER } from "./transport.js"; +import { spawn as nodeSpawn } from 'node:child_process'; +import type { K8sSandboxConfig } from './config.js'; +import type { ExecInPod, SandboxTransport } from './transport.js'; +import { DEFAULT_OUTPUT_CAP, OUTPUT_TRUNCATED_MARKER } from './transport.js'; // Re-export so existing `./exec.js` importers of ExecInPod keep working. -export type { ExecInPod, ExecResult } from "./transport.js"; +export type { ExecInPod, ExecResult } from './transport.js'; type SpawnFn = typeof nodeSpawn; /** Pure argv builder for `kubectl exec` (unit-tested). */ export function buildKubectlArgs(config: K8sSandboxConfig, command: string): string[] { - const args = ["exec", "-i", "-n", config.namespace]; - if (config.context) args.push("--context", config.context); - args.push(config.pod, "--", "bash", "-c", command); + const args = ['exec', '-i', '-n', config.namespace]; + if (config.context) args.push('--context', config.context); + args.push(config.pod, '--', 'bash', '-c', command); return args; } /** True only when exec timing is explicitly enabled (off by default). */ export function shouldEmitExecTiming(env: NodeJS.ProcessEnv): boolean { - return env.KAGENTI_EXEC_TIMING === "1"; + return env.KAGENTI_EXEC_TIMING === '1'; } /** One stable, newline-terminated timing line for a single exec. */ export function formatExecTiming(pod: string, ms: number, command: string): string { - const cmd = command.slice(0, 60).replace(/\s+/g, " "); + const cmd = command.slice(0, 60).replace(/\s+/g, ' '); return `[exec-timing] pod=${pod} ms=${ms} cmd=${cmd}\n`; } @@ -40,8 +40,8 @@ export function KubectlTransport( const outputCap = deps.outputCapBytes ?? DEFAULT_OUTPUT_CAP; const exec: ExecInPod = (command, opts = {}) => new Promise((resolve, reject) => { - const child = spawnFn("kubectl", buildKubectlArgs(config, command), { - stdio: ["pipe", "pipe", "pipe"], + const child = spawnFn('kubectl', buildKubectlArgs(config, command), { + stdio: ['pipe', 'pipe', 'pipe'], }); const startMs = Date.now(); const out: Buffer[] = []; @@ -51,13 +51,13 @@ export function KubectlTransport( opts.timeout && opts.timeout > 0 ? setTimeout(() => { timedOut = true; - child.kill("SIGKILL"); + child.kill('SIGKILL'); }, opts.timeout * 1000) : undefined; let bytes = 0; let truncated = false; - child.stdout.on("data", (d: Buffer) => { + child.stdout.on('data', (d: Buffer) => { opts.onData?.(d); // streaming is uncapped; the cap is on what Pi gets back if (truncated) return; out.push(d); @@ -70,29 +70,30 @@ export function KubectlTransport( // (if at all) by EPIPE on its next write to the now-closed stream — fine for // cat/grep, but a hostile producer that traps or ignores SIGPIPE keeps // running in the pod after this returns (spec §8 "Poisoned-output defense"). - child.kill("SIGKILL"); + child.kill('SIGKILL'); } }); - child.stderr.on("data", (d: Buffer) => { + child.stderr.on('data', (d: Buffer) => { opts.onData?.(d); }); - const onAbort = () => child.kill("SIGKILL"); - opts.signal?.addEventListener("abort", onAbort, { once: true }); + const onAbort = () => child.kill('SIGKILL'); + opts.signal?.addEventListener('abort', onAbort, { once: true }); - child.on("error", (e) => { + child.on('error', (e) => { if (settled) return; settled = true; if (timer) clearTimeout(timer); reject(e); }); - child.on("close", (code) => { + child.on('close', (code) => { if (settled) return; settled = true; if (timer) clearTimeout(timer); - opts.signal?.removeEventListener("abort", onAbort); - if (opts.signal?.aborted) return reject(new Error("aborted")); - if (truncated) return resolve({ stdout: Buffer.concat(out), exitCode: null, truncated: true }); + opts.signal?.removeEventListener('abort', onAbort); + if (opts.signal?.aborted) return reject(new Error('aborted')); + if (truncated) + return resolve({ stdout: Buffer.concat(out), exitCode: null, truncated: true }); if (timedOut) return reject(new Error(`timeout:${opts.timeout}`)); if (shouldEmitExecTiming(process.env)) { process.stderr.write(formatExecTiming(config.pod, Date.now() - startMs, command)); diff --git a/packages/k8s-sandbox/src/extension.ts b/packages/k8s-sandbox/src/extension.ts index 253622a..21d836d 100644 --- a/packages/k8s-sandbox/src/extension.ts +++ b/packages/k8s-sandbox/src/extension.ts @@ -7,11 +7,11 @@ import { createWriteTool, type ExtensionAPI, type ExtensionFactory, -} from "@earendil-works/pi-coding-agent"; -import { type K8sSandboxConfig, resolveConfig } from "./config.js"; -import { KubectlTransport } from "./exec.js"; -import type { SandboxTransport } from "./transport.js"; -import { persistentExecInPod } from "./persistent-exec.js"; +} from '@earendil-works/pi-coding-agent'; +import { type K8sSandboxConfig, resolveConfig } from './config.js'; +import { KubectlTransport } from './exec.js'; +import type { SandboxTransport } from './transport.js'; +import { persistentExecInPod } from './persistent-exec.js'; import { createPodBashOps, createPodEditOps, @@ -19,8 +19,8 @@ import { createPodLsOps, createPodReadOps, createPodWriteOps, -} from "./operations.js"; -import { createPodGrepTool } from "./grep-tool.js"; +} from './operations.js'; +import { createPodGrepTool } from './grep-tool.js'; /** * Pi extension that overrides the seven built-in tools so file/search/exec run @@ -43,8 +43,7 @@ export function k8sSandboxExtension(opts?: { }): ExtensionFactory { return (pi: ExtensionAPI) => { const localCwd = process.cwd(); - const config = - opts?.config !== undefined ? opts.config : resolveConfig(process.env, localCwd); + const config = opts?.config !== undefined ? opts.config : resolveConfig(process.env, localCwd); if (!config) return; // off gate — local tools stand const streamTransport = opts?.transport ?? KubectlTransport(config); @@ -52,19 +51,31 @@ export function k8sSandboxExtension(opts?: { opts?.transport ?? persistentExecInPod(config, { fallback: KubectlTransport(config).exec }); // Fast request/response ops → persistent channel. - pi.registerTool(createReadTool(localCwd, { operations: createPodReadOps(fastTransport.exec, config) })); - pi.registerTool(createWriteTool(localCwd, { operations: createPodWriteOps(fastTransport.exec, config) })); - pi.registerTool(createEditTool(localCwd, { operations: createPodEditOps(fastTransport.exec, config) })); - pi.registerTool(createLsTool(localCwd, { operations: createPodLsOps(fastTransport.exec, config) })); - pi.registerTool(createFindTool(localCwd, { operations: createPodFindOps(fastTransport.exec, config) })); + pi.registerTool( + createReadTool(localCwd, { operations: createPodReadOps(fastTransport.exec, config) }), + ); + pi.registerTool( + createWriteTool(localCwd, { operations: createPodWriteOps(fastTransport.exec, config) }), + ); + pi.registerTool( + createEditTool(localCwd, { operations: createPodEditOps(fastTransport.exec, config) }), + ); + pi.registerTool( + createLsTool(localCwd, { operations: createPodLsOps(fastTransport.exec, config) }), + ); + pi.registerTool( + createFindTool(localCwd, { operations: createPodFindOps(fastTransport.exec, config) }), + ); // Streaming / long-running ops → per-call kubectl exec. - pi.registerTool(createBashTool(localCwd, { operations: createPodBashOps(streamTransport.exec, config) })); + pi.registerTool( + createBashTool(localCwd, { operations: createPodBashOps(streamTransport.exec, config) }), + ); pi.registerTool(createPodGrepTool(localCwd, streamTransport.exec, config)); - pi.on("user_bash", () => ({ operations: createPodBashOps(streamTransport.exec, config) })); + pi.on('user_bash', () => ({ operations: createPodBashOps(streamTransport.exec, config) })); // Tell the model its cwd is the pod's, not the head's. - pi.on("before_agent_start", (event) => { + pi.on('before_agent_start', (event) => { const modified = event.systemPrompt.replace( `Current working directory: ${localCwd}`, `Current working directory: ${config.podCwd} (sandbox pod ${config.namespace}/${config.pod})`, @@ -73,7 +84,7 @@ export function k8sSandboxExtension(opts?: { }); // Tear down the persistent kubectl process so it is never leaked. - pi.on("session_shutdown", async () => { + pi.on('session_shutdown', async () => { await fastTransport.close(); }); }; diff --git a/packages/k8s-sandbox/src/framing.ts b/packages/k8s-sandbox/src/framing.ts index a9cb543..a059117 100644 --- a/packages/k8s-sandbox/src/framing.ts +++ b/packages/k8s-sandbox/src/framing.ts @@ -4,7 +4,7 @@ // base64-encoded. base64's alphabet ([A-Za-z0-9+/=] + "\n") cannot contain the // \x01-prefixed markers, so framing is collision-proof and binary-safe. -const SOH = "\x01"; // marker lead byte; never appears in base64 output +const SOH = '\x01'; // marker lead byte; never appears in base64 output /** * Reported in place of the command's exit code when a stage of the wrapper pipeline @@ -81,13 +81,13 @@ export function wrapCommand( // use a delimiter containing '_', which the standard base64 alphabet // (A-Za-z0-9+/=) never emits, so it can never collide with a body line. const h = `KAGENTI_EOF_${nonce}`; - return `${begin}{ ${command} <<'${h}'\n${stdin.toString("latin1")}\n${h}\n} | ${cap} | base64; ${end}`; + return `${begin}{ ${command} <<'${h}'\n${stdin.toString('latin1')}\n${h}\n} | ${cap} | base64; ${end}`; } return `${begin}{ ${command}; } | ${cap} | base64; ${end}`; } function escapeRe(s: string): string { - return s.replace(/[.*+?^${}()|[\]\\]/g, "\\$&"); + return s.replace(/[.*+?^${}()|[\]\\]/g, '\\$&'); } /** Chunk-fed parser that emits complete frames as bytes arrive. */ @@ -99,7 +99,7 @@ export class FrameParser { const frames: Frame[] = []; for (;;) { // latin1 keeps bytes 1:1 for the marker scan; payload is ASCII base64. - const text = this.buf.toString("latin1"); + const text = this.buf.toString('latin1'); const begin = text.match(/\x01B(\S+)\n/); if (!begin) break; const nonce = begin[1]; @@ -111,7 +111,7 @@ export class FrameParser { const b64 = after.slice(0, end.index!); frames.push({ nonce, - stdout: Buffer.from(b64.replace(/\s/g, ""), "base64"), + stdout: Buffer.from(b64.replace(/\s/g, ''), 'base64'), exitCode: parseInt(end[1], 10), }); this.buf = this.buf.subarray(bodyStart + end.index! + end[0].length); diff --git a/packages/k8s-sandbox/src/grep-tool.ts b/packages/k8s-sandbox/src/grep-tool.ts index 3b6cf07..ba3fbb2 100644 --- a/packages/k8s-sandbox/src/grep-tool.ts +++ b/packages/k8s-sandbox/src/grep-tool.ts @@ -1,8 +1,8 @@ -import { isAbsolute, resolve as resolvePath } from "node:path"; -import { createGrepTool } from "@earendil-works/pi-coding-agent"; -import type { K8sSandboxConfig } from "./config.js"; -import type { ExecInPod } from "./exec.js"; -import { mapPath, shQuote } from "./paths.js"; +import { isAbsolute, resolve as resolvePath } from 'node:path'; +import { createGrepTool } from '@earendil-works/pi-coding-agent'; +import type { K8sSandboxConfig } from './config.js'; +import type { ExecInPod } from './exec.js'; +import { mapPath, shQuote } from './paths.js'; /** * grep cannot be routed via GrepOperations because Pi's grep always spawns a @@ -19,25 +19,25 @@ export function createPodGrepTool( return { ...base, async execute(_id: string, params: Record, signal?: AbortSignal) { - const pattern = String(params.pattern ?? ""); - const searchDir = typeof params.path === "string" ? params.path : "."; - const glob = typeof params.glob === "string" ? params.glob : undefined; + const pattern = String(params.pattern ?? ''); + const searchDir = typeof params.path === 'string' ? params.path : '.'; + const glob = typeof params.glob === 'string' ? params.glob : undefined; const ignoreCase = params.ignoreCase === true; const literal = params.literal === true; - const context = typeof params.context === "number" ? params.context : 0; + const context = typeof params.context === 'number' ? params.context : 0; const headPath = isAbsolute(searchDir) ? searchDir : resolvePath(localCwd, searchDir); const podPath = mapPath(headPath, cfg.headCwd, cfg.podCwd); - const parts = ["rg", "--line-number", "--no-heading", "--color=never", "--hidden"]; - if (ignoreCase) parts.push("--ignore-case"); - if (literal) parts.push("--fixed-strings"); - if (context > 0) parts.push("--context", String(context)); - if (glob) parts.push("--glob", shQuote(glob)); - parts.push("--", shQuote(pattern), shQuote(podPath)); + const parts = ['rg', '--line-number', '--no-heading', '--color=never', '--hidden']; + if (ignoreCase) parts.push('--ignore-case'); + if (literal) parts.push('--fixed-strings'); + if (context > 0) parts.push('--context', String(context)); + if (glob) parts.push('--glob', shQuote(glob)); + parts.push('--', shQuote(pattern), shQuote(podPath)); const streamed: Buffer[] = []; - const r = await exec(parts.join(" "), { + const r = await exec(parts.join(' '), { signal, onData: (chunk) => streamed.push(chunk), }); @@ -49,14 +49,20 @@ export function createPodGrepTool( } if (r.exitCode !== 0 && r.exitCode !== 1) { const detail = Buffer.concat(streamed).toString().trim(); - throw new Error(`rg failed in pod (exit ${r.exitCode})${detail ? `: ${detail}` : ""}`); + throw new Error(`rg failed in pod (exit ${r.exitCode})${detail ? `: ${detail}` : ''}`); } const text = r.stdout.toString(); // rg exits 1 with no output when there are no matches. - if (r.exitCode === 1 && text.trim() === "") { - return { content: [{ type: "text" as const, text: "No matches found" }], details: undefined }; + if (r.exitCode === 1 && text.trim() === '') { + return { + content: [{ type: 'text' as const, text: 'No matches found' }], + details: undefined, + }; } - return { content: [{ type: "text" as const, text: text.length ? text : "No matches found" }], details: undefined }; + return { + content: [{ type: 'text' as const, text: text.length ? text : 'No matches found' }], + details: undefined, + }; }, } as ReturnType; } diff --git a/packages/k8s-sandbox/src/grpc-relay-transport.ts b/packages/k8s-sandbox/src/grpc-relay-transport.ts index 6f4f85e..d079f06 100644 --- a/packages/k8s-sandbox/src/grpc-relay-transport.ts +++ b/packages/k8s-sandbox/src/grpc-relay-transport.ts @@ -1,18 +1,18 @@ -import { DEFAULT_OUTPUT_CAP, OUTPUT_TRUNCATED_MARKER, type SandboxTransport } from "./transport.js"; -import { makeReqIdSource } from "./req-id.js"; +import { DEFAULT_OUTPUT_CAP, OUTPUT_TRUNCATED_MARKER, type SandboxTransport } from './transport.js'; +import { makeReqIdSource } from './req-id.js'; import { Stream, type AbortRequest, type ExecEvent, type ExecRequest, -} from "./gen/sandbox/v1/sandbox.js"; +} from './gen/sandbox/v1/sandbox.js'; /** Minimal surface of the generated SandboxExecClient the transport needs. */ export interface ExecClientLike { exec(request: ExecRequest): { - on(event: "data", cb: (ev: ExecEvent) => void): unknown; - on(event: "end", cb: () => void): unknown; - on(event: "error", cb: (err: Error) => void): unknown; + on(event: 'data', cb: (ev: ExecEvent) => void): unknown; + on(event: 'end', cb: () => void): unknown; + on(event: 'error', cb: (err: Error) => void): unknown; cancel(): void; }; abort(request: AbortRequest, cb: (err: Error | null) => void): unknown; @@ -38,7 +38,7 @@ export function GrpcRelayTransport( const nextReqId = opts.reqIdSource ?? defaultReqIdSource; let closed = false; - const exec: SandboxTransport["exec"] = (command, execOpts = {}) => + const exec: SandboxTransport['exec'] = (command, execOpts = {}) => new Promise((resolve, reject) => { const reqId = nextReqId(); const call = client.exec({ @@ -61,27 +61,32 @@ export function GrpcRelayTransport( if (settled) return; // dedup: drop late frames for a settled reqId settled = true; clearTimeout(timer); - if (execOpts.signal) execOpts.signal.removeEventListener("abort", onAbort); + if (execOpts.signal) execOpts.signal.removeEventListener('abort', onAbort); fn(); }; - const timer = setTimeout(() => { - call.cancel(); - client.abort({ sandboxId, reqId }, () => {}); - finish(() => reject(new Error(`timeout:${execOpts.timeout ?? Math.round(deadlineMs / 1000)}`))); - }, execOpts.timeout ? execOpts.timeout * 1000 : deadlineMs); + const timer = setTimeout( + () => { + call.cancel(); + client.abort({ sandboxId, reqId }, () => {}); + finish(() => + reject(new Error(`timeout:${execOpts.timeout ?? Math.round(deadlineMs / 1000)}`)), + ); + }, + execOpts.timeout ? execOpts.timeout * 1000 : deadlineMs, + ); const onAbort = () => { call.cancel(); client.abort({ sandboxId, reqId }, () => {}); - finish(() => reject(new Error("aborted"))); + finish(() => reject(new Error('aborted'))); }; if (execOpts.signal) { if (execOpts.signal.aborted) return onAbort(); - execOpts.signal.addEventListener("abort", onAbort); + execOpts.signal.addEventListener('abort', onAbort); } - call.on("data", (ev: ExecEvent) => { + call.on('data', (ev: ExecEvent) => { if (settled) return; if (ev.chunk) { const data = Buffer.from(ev.chunk.data); @@ -95,20 +100,26 @@ export function GrpcRelayTransport( stdout.push(Buffer.from(OUTPUT_TRUNCATED_MARKER)); call.cancel(); client.abort({ sandboxId, reqId }, () => {}); - finish(() => resolve({ stdout: Buffer.concat(stdout), exitCode: null, truncated: true })); + finish(() => + resolve({ stdout: Buffer.concat(stdout), exitCode: null, truncated: true }), + ); } } } } else if (ev.end) { const code = ev.end.exitCode < 0 ? null : ev.end.exitCode; - finish(() => resolve({ stdout: Buffer.concat(stdout), exitCode: code, truncated: false })); + finish(() => + resolve({ stdout: Buffer.concat(stdout), exitCode: code, truncated: false }), + ); } else if (ev.error) { finish(() => reject(new Error(ev.error!.message))); } }); - call.on("error", (err: Error) => finish(() => reject(err))); + call.on('error', (err: Error) => finish(() => reject(err))); // Stream ended with no End frame: no exit status, and NOT our cap. - call.on("end", () => finish(() => resolve({ stdout: Buffer.concat(stdout), exitCode: null, truncated: false }))); + call.on('end', () => + finish(() => resolve({ stdout: Buffer.concat(stdout), exitCode: null, truncated: false })), + ); }); return { diff --git a/packages/k8s-sandbox/src/index.ts b/packages/k8s-sandbox/src/index.ts index c005f65..48fedf9 100644 --- a/packages/k8s-sandbox/src/index.ts +++ b/packages/k8s-sandbox/src/index.ts @@ -1,15 +1,34 @@ -export { k8sSandboxExtension } from "./extension.js"; -export { resolveConfig, type K8sSandboxConfig } from "./config.js"; -export { buildKubectlArgs, KubectlTransport, type ExecInPod, type ExecResult } from "./exec.js"; -export type { SandboxTransport } from "./transport.js"; -export { buildPersistentKubectlArgs, persistentExecInPod } from "./persistent-exec.js"; -export { buildSelectorArgs, buildPodNameArgs, resolveSandboxConfig, type RunKubectl } from "./resolve-pod.js"; -export { buildPoolPodsArgs, parsePodNames, listPoolPods } from "./pool.js"; -export { defaultRunKubectl } from "./resolve-pod.js"; -export { GrpcRelayTransport, type ExecClientLike } from "./grpc-relay-transport.js"; +export { k8sSandboxExtension } from './extension.js'; +export { resolveConfig, type K8sSandboxConfig } from './config.js'; +export { buildKubectlArgs, KubectlTransport, type ExecInPod, type ExecResult } from './exec.js'; +export type { SandboxTransport } from './transport.js'; +export { buildPersistentKubectlArgs, persistentExecInPod } from './persistent-exec.js'; export { - WorkerFrame, ServerFrame, Hello, Exec, Abort, Chunk, End, ExecError, - ExecEvent, ExecRequest, AbortRequest, AbortResponse, Stream, - SandboxWorkerService, SandboxExecService, SandboxExecClient, - type SandboxWorkerServer, type SandboxExecServer, -} from "./gen/sandbox/v1/sandbox.js"; + buildSelectorArgs, + buildPodNameArgs, + resolveSandboxConfig, + type RunKubectl, +} from './resolve-pod.js'; +export { buildPoolPodsArgs, parsePodNames, listPoolPods } from './pool.js'; +export { defaultRunKubectl } from './resolve-pod.js'; +export { GrpcRelayTransport, type ExecClientLike } from './grpc-relay-transport.js'; +export { + WorkerFrame, + ServerFrame, + Hello, + Exec, + Abort, + Chunk, + End, + ExecError, + ExecEvent, + ExecRequest, + AbortRequest, + AbortResponse, + Stream, + SandboxWorkerService, + SandboxExecService, + SandboxExecClient, + type SandboxWorkerServer, + type SandboxExecServer, +} from './gen/sandbox/v1/sandbox.js'; diff --git a/packages/k8s-sandbox/src/operations.ts b/packages/k8s-sandbox/src/operations.ts index c194496..bc8025c 100644 --- a/packages/k8s-sandbox/src/operations.ts +++ b/packages/k8s-sandbox/src/operations.ts @@ -5,13 +5,13 @@ import type { LsOperations, ReadOperations, WriteOperations, -} from "@earendil-works/pi-coding-agent"; -import type { K8sSandboxConfig } from "./config.js"; -import type { ExecInPod } from "./exec.js"; -import { mapPath, shQuote } from "./paths.js"; -import { DEFAULT_OUTPUT_CAP } from "./transport.js"; +} from '@earendil-works/pi-coding-agent'; +import type { K8sSandboxConfig } from './config.js'; +import type { ExecInPod } from './exec.js'; +import { mapPath, shQuote } from './paths.js'; +import { DEFAULT_OUTPUT_CAP } from './transport.js'; -const IMAGE_MIMES = ["image/jpeg", "image/png", "image/gif", "image/webp"]; +const IMAGE_MIMES = ['image/jpeg', 'image/png', 'image/gif', 'image/webp']; function mapper(cfg: K8sSandboxConfig) { return (p: string) => shQuote(mapPath(p, cfg.headCwd, cfg.podCwd)); @@ -35,7 +35,7 @@ export function createPodReadOps(exec: ExecInPod, cfg: K8sSandboxConfig): ReadOp const size = stat && stat.exitCode === 0 ? stat.stdout.toString().trim() : null; throw new Error( `Read exceeds the ${DEFAULT_OUTPUT_CAP} byte sandbox output cap` + - `${size ? ` (file is ${size} bytes)` : ""}: ${p}. ` + + `${size ? ` (file is ${size} bytes)` : ''}: ${p}. ` + `Read a range with bash instead, e.g. sed -n '1,500p' .`, ); } @@ -63,7 +63,7 @@ export function createPodWriteOps(exec: ExecInPod, cfg: K8sSandboxConfig): Write const q = mapper(cfg); return { writeFile: async (p, content) => { - const b64 = Buffer.from(content).toString("base64"); + const b64 = Buffer.from(content).toString('base64'); const r = await exec(`base64 -d > ${q(p)}`, { stdin: Buffer.from(b64) }); if (r.exitCode !== 0) throw new Error(`Write failed in pod: ${p}`); }, @@ -103,7 +103,7 @@ export function createPodBashOps(exec: ExecInPod, cfg: K8sSandboxConfig): BashOp .map(([k, v]) => `${k}=${shQuote(String(v))}`) : []; const wrapped = pairs.length - ? `cd ${q(cwd)} && env ${pairs.join(" ")} bash -c ${shQuote(command)}` + ? `cd ${q(cwd)} && env ${pairs.join(' ')} bash -c ${shQuote(command)}` : `cd ${q(cwd)} && ${command}`; // M2's exact form — unchanged when no env const r = await exec(wrapped, { onData, signal, timeout }); // A cap trip means the command was SIGKILLed mid-flight with no exit status. Pi @@ -129,7 +129,7 @@ export function createPodLsOps(exec: ExecInPod, cfg: K8sSandboxConfig): LsOperat stat: async (p) => { const r = await exec(`test -e ${q(p)} && (test -d ${q(p)} && echo DIR || echo FILE)`); if (r.exitCode !== 0) throw new Error(`Path not found in pod: ${p}`); - const isDir = r.stdout.toString().trim() === "DIR"; + const isDir = r.stdout.toString().trim() === 'DIR'; return { isDirectory: () => isDir }; }, readdir: async (p) => { @@ -144,7 +144,10 @@ export function createPodLsOps(exec: ExecInPod, cfg: K8sSandboxConfig): LsOperat ); } if (r.exitCode !== 0) throw new Error(`readdir failed in pod: ${p}`); - return r.stdout.toString().split("\n").filter((x) => x.length > 0); + return r.stdout + .toString() + .split('\n') + .filter((x) => x.length > 0); }, }; } @@ -164,7 +167,7 @@ export function createPodFindOps(exec: ExecInPod, cfg: K8sSandboxConfig): FindOp glob: async (pattern, cwd, { ignore, limit }) => { const globs = [`-g ${shQuote(pattern)}`, ...ignore.map((ig) => `-g ${shQuote(`!${ig}`)}`)]; const r = await exec( - `cd ${q(cwd)} && rg --files --hidden ${globs.join(" ")} | head -n ${limit}; ` + + `cd ${q(cwd)} && rg --files --hidden ${globs.join(' ')} | head -n ${limit}; ` + `rc=\${PIPESTATUS[0]}; [ "\$rc" = 0 ] || [ "\$rc" = 1 ] || [ "\$rc" = 141 ] || exit "\$rc"`, ); // Two independent concerns, checked in order. @@ -193,9 +196,9 @@ export function createPodFindOps(exec: ExecInPod, cfg: K8sSandboxConfig): FindOp } return r.stdout .toString() - .split("\n") + .split('\n') .filter((x) => x.length > 0) - .map((rel) => rel.replace(/^\.\//, "")); + .map((rel) => rel.replace(/^\.\//, '')); }, }; } diff --git a/packages/k8s-sandbox/src/paths.ts b/packages/k8s-sandbox/src/paths.ts index e2e1357..8ef2496 100644 --- a/packages/k8s-sandbox/src/paths.ts +++ b/packages/k8s-sandbox/src/paths.ts @@ -10,6 +10,6 @@ export function shQuote(s: string): string { */ export function mapPath(p: string, headCwd: string, podCwd: string): string { if (p === headCwd) return podCwd; - if (p.startsWith(headCwd + "/")) return podCwd + p.slice(headCwd.length); + if (p.startsWith(headCwd + '/')) return podCwd + p.slice(headCwd.length); return p; } diff --git a/packages/k8s-sandbox/src/persistent-exec.ts b/packages/k8s-sandbox/src/persistent-exec.ts index 01e7f6a..a2c3927 100644 --- a/packages/k8s-sandbox/src/persistent-exec.ts +++ b/packages/k8s-sandbox/src/persistent-exec.ts @@ -1,14 +1,14 @@ -import { type ChildProcess, spawn as nodeSpawn } from "node:child_process"; -import type { K8sSandboxConfig } from "./config.js"; -import type { ExecInPod, SandboxTransport } from "./transport.js"; -import { DEFAULT_OUTPUT_CAP, OUTPUT_TRUNCATED_MARKER } from "./transport.js"; -import { CAP_STAGE_FAILED, FrameParser, wrapCommand } from "./framing.js"; +import { type ChildProcess, spawn as nodeSpawn } from 'node:child_process'; +import type { K8sSandboxConfig } from './config.js'; +import type { ExecInPod, SandboxTransport } from './transport.js'; +import { DEFAULT_OUTPUT_CAP, OUTPUT_TRUNCATED_MARKER } from './transport.js'; +import { CAP_STAGE_FAILED, FrameParser, wrapCommand } from './framing.js'; /** argv for the long-lived session: a bare interactive `bash` (NOT `bash -c`). */ export function buildPersistentKubectlArgs(config: K8sSandboxConfig): string[] { - const args = ["exec", "-i", "-n", config.namespace]; - if (config.context) args.push("--context", config.context); - args.push(config.pod, "--", "bash"); + const args = ['exec', '-i', '-n', config.namespace]; + if (config.context) args.push('--context', config.context); + args.push(config.pod, '--', 'bash'); return args; } @@ -54,7 +54,7 @@ export function persistentExecInPod( const c = child; child = null; try { - c.kill("SIGKILL"); + c.kill('SIGKILL'); } catch { /* already gone */ } @@ -77,11 +77,11 @@ export function persistentExecInPod( const ensureChild = () => { if (child || disposed) return; - const c = spawnFn("kubectl", buildPersistentKubectlArgs(config), { - stdio: ["pipe", "pipe", "pipe"], + const c = spawnFn('kubectl', buildPersistentKubectlArgs(config), { + stdio: ['pipe', 'pipe', 'pipe'], }); child = c; - c.stdout!.on("data", (d: Buffer) => { + c.stdout!.on('data', (d: Buffer) => { for (const f of parser.push(d)) { if (inflight && f.nonce === inflight.nonce) { const cur = inflight; @@ -114,13 +114,16 @@ export function persistentExecInPod( ); killChild(); parser = new FrameParser(); - cur.fail(new Error("persistent channel output-cap stage unavailable")); + cur.fail(new Error('persistent channel output-cap stage unavailable')); pump(); continue; } if (f.stdout.length > outputCap) { cur.done({ - stdout: Buffer.concat([f.stdout.subarray(0, outputCap), Buffer.from(OUTPUT_TRUNCATED_MARKER)]), + stdout: Buffer.concat([ + f.stdout.subarray(0, outputCap), + Buffer.from(OUTPUT_TRUNCATED_MARKER), + ]), exitCode: null, truncated: true, }); @@ -131,9 +134,9 @@ export function persistentExecInPod( } } }); - c.on("error", (e) => failSession(e instanceof Error ? e : new Error(String(e)))); - c.on("close", () => { - if (inflight || queue.length) failSession(new Error("session closed")); + c.on('error', (e) => failSession(e instanceof Error ? e : new Error(String(e)))); + c.on('close', () => { + if (inflight || queue.length) failSession(new Error('session closed')); else child = null; }); }; @@ -153,7 +156,7 @@ export function persistentExecInPod( let timer: ReturnType | undefined; const cleanup = () => { if (timer) clearTimeout(timer); - opts.signal?.removeEventListener("abort", onAbort); + opts.signal?.removeEventListener('abort', onAbort); }; // timeout / abort: kill+reset the session and reject (NO fallback). const killAndReject = (err: Error) => { @@ -167,12 +170,15 @@ export function persistentExecInPod( pump(); }; function onAbort() { - killAndReject(new Error("aborted")); + killAndReject(new Error('aborted')); } - if (opts.signal?.aborted) return killAndReject(new Error("aborted")); - opts.signal?.addEventListener("abort", onAbort, { once: true }); + if (opts.signal?.aborted) return killAndReject(new Error('aborted')); + opts.signal?.addEventListener('abort', onAbort, { once: true }); if (opts.timeout && opts.timeout > 0) { - timer = setTimeout(() => killAndReject(new Error(`timeout:${opts.timeout}`)), opts.timeout * 1000); + timer = setTimeout( + () => killAndReject(new Error(`timeout:${opts.timeout}`)), + opts.timeout * 1000, + ); } inflight = { nonce, diff --git a/packages/k8s-sandbox/src/pool.ts b/packages/k8s-sandbox/src/pool.ts index dbf55e2..73aa3db 100644 --- a/packages/k8s-sandbox/src/pool.ts +++ b/packages/k8s-sandbox/src/pool.ts @@ -1,16 +1,27 @@ -import { type RunKubectl, defaultRunKubectl } from "./resolve-pod.js"; +import { type RunKubectl, defaultRunKubectl } from './resolve-pod.js'; /** Pure: kubectl args to list Running pod names matching a label selector (one name per line). */ export function buildPoolPodsArgs(selector: string, namespace: string, context?: string): string[] { - const args = ["get", "pod", "-n", namespace, "-l", selector, "--field-selector=status.phase=Running"]; - if (context) args.push("--context", context); - args.push("-o", "jsonpath={range .items[*]}{.metadata.name}{'\\n'}{end}"); + const args = [ + 'get', + 'pod', + '-n', + namespace, + '-l', + selector, + '--field-selector=status.phase=Running', + ]; + if (context) args.push('--context', context); + args.push('-o', "jsonpath={range .items[*]}{.metadata.name}{'\\n'}{end}"); return args; } /** Pure: parse newline-separated pod names from kubectl stdout. */ export function parsePodNames(stdout: string): string[] { - return stdout.split("\n").map((s) => s.trim()).filter(Boolean); + return stdout + .split('\n') + .map((s) => s.trim()) + .filter(Boolean); } /** List Running pod names in the pool (all pods matching the shared pool label). */ diff --git a/packages/k8s-sandbox/src/req-id.ts b/packages/k8s-sandbox/src/req-id.ts index 75f3017..5a3b92a 100644 --- a/packages/k8s-sandbox/src/req-id.ts +++ b/packages/k8s-sandbox/src/req-id.ts @@ -1,4 +1,4 @@ -import { randomInt } from "node:crypto"; +import { randomInt } from 'node:crypto'; const SALT_BITS = 21; // 2^21 ≈ 2.1M distinct replica spaces const COUNTER_SPACE = 2 ** 32; diff --git a/packages/k8s-sandbox/src/resolve-pod.ts b/packages/k8s-sandbox/src/resolve-pod.ts index cde0dc8..f3ef1bc 100644 --- a/packages/k8s-sandbox/src/resolve-pod.ts +++ b/packages/k8s-sandbox/src/resolve-pod.ts @@ -1,19 +1,27 @@ -import { spawn } from "node:child_process"; -import { resolveConfig, type K8sSandboxConfig } from "./config.js"; +import { spawn } from 'node:child_process'; +import { resolveConfig, type K8sSandboxConfig } from './config.js'; /** Pure: kubectl args to read a Sandbox's status.selector (a label-selector string). */ export function buildSelectorArgs(name: string, namespace: string, context?: string): string[] { - const args = ["get", "sandbox", name, "-n", namespace]; - if (context) args.push("--context", context); - args.push("-o", "jsonpath={.status.selector}"); + const args = ['get', 'sandbox', name, '-n', namespace]; + if (context) args.push('--context', context); + args.push('-o', 'jsonpath={.status.selector}'); return args; } /** Pure: kubectl args to read the first Running pod name matching a label selector. */ export function buildPodNameArgs(selector: string, namespace: string, context?: string): string[] { - const args = ["get", "pod", "-n", namespace, "-l", selector, "--field-selector=status.phase=Running"]; - if (context) args.push("--context", context); - args.push("-o", "jsonpath={.items[0].metadata.name}"); + const args = [ + 'get', + 'pod', + '-n', + namespace, + '-l', + selector, + '--field-selector=status.phase=Running', + ]; + if (context) args.push('--context', context); + args.push('-o', 'jsonpath={.items[0].metadata.name}'); return args; } @@ -21,16 +29,20 @@ export type RunKubectl = (args: string[]) => Promise; export const defaultRunKubectl: RunKubectl = (args) => new Promise((resolve, reject) => { - const child = spawn("kubectl", args, { stdio: ["ignore", "pipe", "pipe"] }); + const child = spawn('kubectl', args, { stdio: ['ignore', 'pipe', 'pipe'] }); const out: Buffer[] = []; const err: Buffer[] = []; - child.stdout.on("data", (d: Buffer) => out.push(d)); - child.stderr.on("data", (d: Buffer) => err.push(d)); - child.on("error", reject); - child.on("close", (code) => + child.stdout.on('data', (d: Buffer) => out.push(d)); + child.stderr.on('data', (d: Buffer) => err.push(d)); + child.on('error', reject); + child.on('close', (code) => code === 0 ? resolve(Buffer.concat(out).toString().trim()) - : reject(new Error(`kubectl ${args.join(" ")} failed (${code}): ${Buffer.concat(err).toString().trim()}`)), + : reject( + new Error( + `kubectl ${args.join(' ')} failed (${code}): ${Buffer.concat(err).toString().trim()}`, + ), + ), ); }); @@ -53,7 +65,7 @@ export async function resolveSandboxConfig( const name = env.KAGENTI_SANDBOX_NAME; if (!name) return null; - const namespace = env.KAGENTI_SANDBOX_NAMESPACE ?? "default"; + const namespace = env.KAGENTI_SANDBOX_NAMESPACE ?? 'default'; const context = env.KAGENTI_SANDBOX_CONTEXT || undefined; const selector = (await run(buildSelectorArgs(name, namespace, context))).trim(); @@ -61,5 +73,5 @@ export async function resolveSandboxConfig( const pod = (await run(buildPodNameArgs(selector, namespace, context))).trim(); if (!pod) throw new Error(`no Running pod for selector '${selector}'`); - return { pod, namespace, context, podCwd: env.KAGENTI_SANDBOX_CWD ?? "/workspace", headCwd }; + return { pod, namespace, context, podCwd: env.KAGENTI_SANDBOX_CWD ?? '/workspace', headCwd }; } diff --git a/packages/k8s-sandbox/src/transport.ts b/packages/k8s-sandbox/src/transport.ts index 7c4f1ab..39e45d8 100644 --- a/packages/k8s-sandbox/src/transport.ts +++ b/packages/k8s-sandbox/src/transport.ts @@ -73,4 +73,4 @@ export interface SandboxTransport { export const DEFAULT_OUTPUT_CAP = 8 * 1024 * 1024; // 8 MiB /** Appended to returned stdout when the cap trips, so Pi sees the truncation. */ -export const OUTPUT_TRUNCATED_MARKER = "\n[output truncated]"; +export const OUTPUT_TRUNCATED_MARKER = '\n[output truncated]'; diff --git a/packages/k8s-sandbox/test/config.test.ts b/packages/k8s-sandbox/test/config.test.ts index 4ba2af5..f523856 100644 --- a/packages/k8s-sandbox/test/config.test.ts +++ b/packages/k8s-sandbox/test/config.test.ts @@ -1,38 +1,38 @@ -import { describe, expect, it } from "vitest"; -import { resolveConfig } from "../src/config.js"; +import { describe, expect, it } from 'vitest'; +import { resolveConfig } from '../src/config.js'; -describe("resolveConfig", () => { - it("returns null when KAGENTI_SANDBOX_POD is unset (the off gate)", () => { - expect(resolveConfig({}, "/head")).toBeNull(); +describe('resolveConfig', () => { + it('returns null when KAGENTI_SANDBOX_POD is unset (the off gate)', () => { + expect(resolveConfig({}, '/head')).toBeNull(); }); - it("applies defaults when only the pod is set", () => { - const cfg = resolveConfig({ KAGENTI_SANDBOX_POD: "sbx-0" }, "/head"); + it('applies defaults when only the pod is set', () => { + const cfg = resolveConfig({ KAGENTI_SANDBOX_POD: 'sbx-0' }, '/head'); expect(cfg).toEqual({ - pod: "sbx-0", - namespace: "default", + pod: 'sbx-0', + namespace: 'default', context: undefined, - podCwd: "/workspace", - headCwd: "/head", + podCwd: '/workspace', + headCwd: '/head', }); }); - it("honours all overrides", () => { + it('honours all overrides', () => { const cfg = resolveConfig( { - KAGENTI_SANDBOX_POD: "sbx-0", - KAGENTI_SANDBOX_NAMESPACE: "team1", - KAGENTI_SANDBOX_CONTEXT: "kind-kagenti", - KAGENTI_SANDBOX_CWD: "/repo", + KAGENTI_SANDBOX_POD: 'sbx-0', + KAGENTI_SANDBOX_NAMESPACE: 'team1', + KAGENTI_SANDBOX_CONTEXT: 'kind-kagenti', + KAGENTI_SANDBOX_CWD: '/repo', }, - "/head", + '/head', ); expect(cfg).toEqual({ - pod: "sbx-0", - namespace: "team1", - context: "kind-kagenti", - podCwd: "/repo", - headCwd: "/head", + pod: 'sbx-0', + namespace: 'team1', + context: 'kind-kagenti', + podCwd: '/repo', + headCwd: '/head', }); }); }); diff --git a/packages/k8s-sandbox/test/conformance.ts b/packages/k8s-sandbox/test/conformance.ts index c0ac97f..89938c2 100644 --- a/packages/k8s-sandbox/test/conformance.ts +++ b/packages/k8s-sandbox/test/conformance.ts @@ -1,5 +1,5 @@ -import { describe, expect, it, vi, afterEach } from "vitest"; -import { OUTPUT_TRUNCATED_MARKER, type SandboxTransport } from "../src/transport.js"; +import { describe, expect, it, vi, afterEach } from 'vitest'; +import { OUTPUT_TRUNCATED_MARKER, type SandboxTransport } from '../src/transport.js'; /** * A scripted sandbox backend, transport-agnostic. Each transport's conformance @@ -38,7 +38,7 @@ export interface FakeBehavior { * a regression that deletes the stop fails loudly instead of * silently reporting `true`. */ -export type ProducerStop = "remote-abort" | "local-kill" | "producer-side-cap" | "none"; +export type ProducerStop = 'remote-abort' | 'local-kill' | 'producer-side-cap' | 'none'; /** What a transport declares about itself to the battery. */ export interface TransportCapabilities { @@ -84,47 +84,47 @@ export function runConformance( afterEach(() => vi.useRealTimers()); describe(`SandboxTransport conformance: ${label}`, () => { - it("returns collected stdout and the exit code", async () => { - const { transport } = make({ stdout: ["foo", "bar"], exitCode: 0 }); - const r = await transport.exec("echo hi"); - expect(r.stdout.toString()).toBe("foobar"); + it('returns collected stdout and the exit code', async () => { + const { transport } = make({ stdout: ['foo', 'bar'], exitCode: 0 }); + const r = await transport.exec('echo hi'); + expect(r.stdout.toString()).toBe('foobar'); expect(r.exitCode).toBe(0); expect(r.truncated).toBe(false); // an untruncated exec must say so explicitly }); it.skipIf(!caps.streams)( - "streams stdout and stderr to onData; stderr is excluded from stdout", + 'streams stdout and stderr to onData; stderr is excluded from stdout', async () => { - const { transport } = make({ stdout: ["out"], stderr: ["err"], exitCode: 0 }); + const { transport } = make({ stdout: ['out'], stderr: ['err'], exitCode: 0 }); const chunks: string[] = []; - const r = await transport.exec("cmd", { onData: (c) => chunks.push(c.toString()) }); - expect(r.stdout.toString()).toBe("out"); // stderr NOT collected - expect(chunks).toContain("out"); - expect(chunks).toContain("err"); // stderr streamed + const r = await transport.exec('cmd', { onData: (c) => chunks.push(c.toString()) }); + expect(r.stdout.toString()).toBe('out'); // stderr NOT collected + expect(chunks).toContain('out'); + expect(chunks).toContain('err'); // stderr streamed }, ); - it("forwards stdin to the backend", async () => { + it('forwards stdin to the backend', async () => { const { transport, stdinSeen } = make({ stdout: [], exitCode: 0 }); - await transport.exec("base64 -d", { stdin: Buffer.from("payload") }); - expect(stdinSeen()?.toString()).toBe("payload"); + await transport.exec('base64 -d', { stdin: Buffer.from('payload') }); + expect(stdinSeen()?.toString()).toBe('payload'); }); - it("propagates a non-zero exit code", async () => { + it('propagates a non-zero exit code', async () => { const { transport } = make({ stdout: [], exitCode: 3 }); - const r = await transport.exec("false"); + const r = await transport.exec('false'); expect(r.exitCode).toBe(3); }); - it("caps returned stdout, appends the truncation marker, and stops collecting", async () => { + it('caps returned stdout, appends the truncation marker, and stops collecting', async () => { // All three transports advertise a total-output-per-exec cap (spec §8): the // concrete mitigation for a hostile sandbox flooding the model's context. A cap on // fewer than all transports is a divergence in the seam, not an implementation detail. - const handle = make({ stdout: ["aaaa", "bbbb", "cccc"], exitCode: 0 }, { outputCapBytes: 6 }); - const r = await handle.transport.exec("cat big"); + const handle = make({ stdout: ['aaaa', 'bbbb', 'cccc'], exitCode: 0 }, { outputCapBytes: 6 }); + const r = await handle.transport.exec('cat big'); const s = r.stdout.toString(); expect(s).toContain(OUTPUT_TRUNCATED_MARKER); - expect(s).not.toContain("cccc"); // collection stopped at the cap + expect(s).not.toContain('cccc'); // collection stopped at the cap expect(r.exitCode).toBeNull(); // the exec was cut short, so there is no real exit code // The seam represents truncation explicitly rather than overloading a null exit // code, which also means "signalled, no status" (spec §3.1). Callers cannot tell @@ -139,23 +139,23 @@ export function runConformance( expect(handle.producerStop()).toBe(caps.producerStop); }); - it("does not flag output that lands exactly on the cap", async () => { + it('does not flag output that lands exactly on the cap', async () => { // The boundary is `> cap`, not `>= cap`. Output that lands exactly on the cap is // COMPLETE and must not be reported as truncated — otherwise every read of a // cap-sized file would fail. The over-cap case above pins the other side. - const { transport } = make({ stdout: ["aaaa", "bb"], exitCode: 0 }, { outputCapBytes: 6 }); - const r = await transport.exec("cat exactly-at-cap"); + const { transport } = make({ stdout: ['aaaa', 'bb'], exitCode: 0 }, { outputCapBytes: 6 }); + const r = await transport.exec('cat exactly-at-cap'); expect(r.truncated).toBe(false); expect(r.exitCode).toBe(0); - expect(r.stdout.toString()).toBe("aaaabb"); + expect(r.stdout.toString()).toBe('aaaabb'); expect(r.stdout.toString()).not.toContain(OUTPUT_TRUNCATED_MARKER); }); - it("rejects with timeout: when the command exceeds the timeout", async () => { + it('rejects with timeout: when the command exceeds the timeout', async () => { vi.useFakeTimers(); const { transport } = make({ hang: true }); - const p = transport.exec("sleep 999", { timeout: 2 }); - const assertion = expect(p).rejects.toThrow("timeout:2"); + const p = transport.exec('sleep 999', { timeout: 2 }); + const assertion = expect(p).rejects.toThrow('timeout:2'); await vi.advanceTimersByTimeAsync(2000); await assertion; }); @@ -163,12 +163,12 @@ export function runConformance( it("rejects with 'aborted' when the signal fires", async () => { const { transport } = make({ hang: true }); const ac = new AbortController(); - const p = transport.exec("sleep 999", { signal: ac.signal }); + const p = transport.exec('sleep 999', { signal: ac.signal }); ac.abort(); - await expect(p).rejects.toThrow("aborted"); + await expect(p).rejects.toThrow('aborted'); }); - it("close() resolves and is idempotent", async () => { + it('close() resolves and is idempotent', async () => { const { transport } = make({ stdout: [], exitCode: 0 }); await expect(transport.close()).resolves.toBeUndefined(); await expect(transport.close()).resolves.toBeUndefined(); diff --git a/packages/k8s-sandbox/test/exec.test.ts b/packages/k8s-sandbox/test/exec.test.ts index 5b50090..4a54449 100644 --- a/packages/k8s-sandbox/test/exec.test.ts +++ b/packages/k8s-sandbox/test/exec.test.ts @@ -1,42 +1,60 @@ -import { describe, expect, it } from "vitest"; -import { buildKubectlArgs, shouldEmitExecTiming, formatExecTiming } from "../src/exec.js"; -import type { K8sSandboxConfig } from "../src/config.js"; +import { describe, expect, it } from 'vitest'; +import { buildKubectlArgs, shouldEmitExecTiming, formatExecTiming } from '../src/exec.js'; +import type { K8sSandboxConfig } from '../src/config.js'; const base: K8sSandboxConfig = { - pod: "sbx-0", - namespace: "team1", + pod: 'sbx-0', + namespace: 'team1', context: undefined, - podCwd: "/workspace", - headCwd: "/head", + podCwd: '/workspace', + headCwd: '/head', }; -describe("buildKubectlArgs", () => { - it("builds an interactive exec with namespace and bash -c", () => { +describe('buildKubectlArgs', () => { + it('builds an interactive exec with namespace and bash -c', () => { expect(buildKubectlArgs(base, "cat '/workspace/a.txt'")).toEqual([ - "exec", "-i", "-n", "team1", "sbx-0", "--", "bash", "-c", "cat '/workspace/a.txt'", + 'exec', + '-i', + '-n', + 'team1', + 'sbx-0', + '--', + 'bash', + '-c', + "cat '/workspace/a.txt'", ]); }); - it("includes --context when set", () => { - expect(buildKubectlArgs({ ...base, context: "kind-kagenti" }, "true")).toEqual([ - "exec", "-i", "-n", "team1", "--context", "kind-kagenti", "sbx-0", "--", "bash", "-c", "true", + it('includes --context when set', () => { + expect(buildKubectlArgs({ ...base, context: 'kind-kagenti' }, 'true')).toEqual([ + 'exec', + '-i', + '-n', + 'team1', + '--context', + 'kind-kagenti', + 'sbx-0', + '--', + 'bash', + '-c', + 'true', ]); }); }); -describe("exec timing (env-gated)", () => { - it("is off unless KAGENTI_EXEC_TIMING=1", () => { +describe('exec timing (env-gated)', () => { + it('is off unless KAGENTI_EXEC_TIMING=1', () => { expect(shouldEmitExecTiming({})).toBe(false); - expect(shouldEmitExecTiming({ KAGENTI_EXEC_TIMING: "0" })).toBe(false); - expect(shouldEmitExecTiming({ KAGENTI_EXEC_TIMING: "1" })).toBe(true); + expect(shouldEmitExecTiming({ KAGENTI_EXEC_TIMING: '0' })).toBe(false); + expect(shouldEmitExecTiming({ KAGENTI_EXEC_TIMING: '1' })).toBe(true); }); - it("formats a single stable line, truncating and flattening the command", () => { - const line = formatExecTiming("sandbox-1", 42, "git -C /workspace/repo fetch\norigin branch-0"); + it('formats a single stable line, truncating and flattening the command', () => { + const line = formatExecTiming('sandbox-1', 42, 'git -C /workspace/repo fetch\norigin branch-0'); expect(line).toBe( - "[exec-timing] pod=sandbox-1 ms=42 cmd=git -C /workspace/repo fetch origin branch-0\n", + '[exec-timing] pod=sandbox-1 ms=42 cmd=git -C /workspace/repo fetch origin branch-0\n', ); - const long = formatExecTiming("p", 1, "x".repeat(200)); - expect(long).toBe(`[exec-timing] pod=p ms=1 cmd=${"x".repeat(60)}\n`); + const long = formatExecTiming('p', 1, 'x'.repeat(200)); + expect(long).toBe(`[exec-timing] pod=p ms=1 cmd=${'x'.repeat(60)}\n`); }); }); diff --git a/packages/k8s-sandbox/test/extension.test.ts b/packages/k8s-sandbox/test/extension.test.ts index 6421b2c..77d280d 100644 --- a/packages/k8s-sandbox/test/extension.test.ts +++ b/packages/k8s-sandbox/test/extension.test.ts @@ -1,14 +1,14 @@ -import { describe, expect, it, vi } from "vitest"; -import type { K8sSandboxConfig } from "../src/config.js"; -import type { SandboxTransport } from "../src/transport.js"; -import { k8sSandboxExtension } from "../src/extension.js"; +import { describe, expect, it, vi } from 'vitest'; +import type { K8sSandboxConfig } from '../src/config.js'; +import type { SandboxTransport } from '../src/transport.js'; +import { k8sSandboxExtension } from '../src/extension.js'; const cfg: K8sSandboxConfig = { - pod: "sbx-0", - namespace: "default", + pod: 'sbx-0', + namespace: 'default', context: undefined, - podCwd: "/workspace", - headCwd: "/head", + podCwd: '/workspace', + headCwd: '/head', }; /** Minimal ExtensionAPI stub recording registrations + event handlers. */ @@ -25,21 +25,21 @@ function fakePi() { } const okTransport = (close = vi.fn(async () => {})): SandboxTransport => ({ - exec: async () => ({ stdout: Buffer.from(""), exitCode: 0, truncated: false }), + exec: async () => ({ stdout: Buffer.from(''), exitCode: 0, truncated: false }), close, }); -describe("k8sSandboxExtension", () => { - it("registers the seven pod tools and wires the lifecycle handlers", () => { +describe('k8sSandboxExtension', () => { + it('registers the seven pod tools and wires the lifecycle handlers', () => { const { pi, tools, handlers } = fakePi(); k8sSandboxExtension({ config: cfg, transport: okTransport() })(pi); expect(tools).toHaveLength(7); - expect(typeof handlers.user_bash).toBe("function"); - expect(typeof handlers.before_agent_start).toBe("function"); - expect(typeof handlers.session_shutdown).toBe("function"); + expect(typeof handlers.user_bash).toBe('function'); + expect(typeof handlers.before_agent_start).toBe('function'); + expect(typeof handlers.session_shutdown).toBe('function'); }); - it("closes the fast channel on session_shutdown", async () => { + it('closes the fast channel on session_shutdown', async () => { const close = vi.fn(async () => {}); const { pi, handlers } = fakePi(); k8sSandboxExtension({ config: cfg, transport: okTransport(close) })(pi); @@ -47,7 +47,7 @@ describe("k8sSandboxExtension", () => { expect(close).toHaveBeenCalledTimes(1); }); - it("is inert (registers nothing) when config is null", () => { + it('is inert (registers nothing) when config is null', () => { const { pi, tools } = fakePi(); k8sSandboxExtension({ config: null })(pi); expect(tools).toHaveLength(0); diff --git a/packages/k8s-sandbox/test/framing.test.ts b/packages/k8s-sandbox/test/framing.test.ts index 333ba17..ff7da1a 100644 --- a/packages/k8s-sandbox/test/framing.test.ts +++ b/packages/k8s-sandbox/test/framing.test.ts @@ -1,20 +1,20 @@ -import { spawnSync } from "node:child_process"; -import { existsSync, mkdtempSync, readFileSync, rmSync, writeFileSync } from "node:fs"; -import { tmpdir } from "node:os"; -import { join } from "node:path"; -import { describe, expect, it } from "vitest"; -import { CAP_STAGE_FAILED, FrameParser, wrapCommand } from "../src/framing.js"; +import { spawnSync } from 'node:child_process'; +import { existsSync, mkdtempSync, readFileSync, rmSync, writeFileSync } from 'node:fs'; +import { tmpdir } from 'node:os'; +import { join } from 'node:path'; +import { describe, expect, it } from 'vitest'; +import { CAP_STAGE_FAILED, FrameParser, wrapCommand } from '../src/framing.js'; -const SOH = "\x01"; +const SOH = '\x01'; -describe("wrapCommand", () => { +describe('wrapCommand', () => { // The cap is applied IN THE POD, on the raw stream before base64. Counting bytes // client-side would cap content at cap × 3/4 (base64 inflation), so the trip point // would differ from the per-call transports — a weaker form of exactly the // distinguishability #180 is about. `head -c` also bounds FrameParser.push's // quadratic buffer growth, and keeps PIPESTATUS[0] indexing the command. it("brackets base64 output with nonce markers, caps raw bytes, and reports the command's exit code", () => { - const line = wrapCommand("n1", "cat '/workspace/a.txt'", undefined, 8); + const line = wrapCommand('n1', "cat '/workspace/a.txt'", undefined, 8); expect(line).toBe( `printf '${SOH}B%s\\n' n1; { cat '/workspace/a.txt'; } | head -c 9 | base64; ` + `st=("\${PIPESTATUS[@]}"); rc="\${st[0]}"; ` + @@ -23,8 +23,8 @@ describe("wrapCommand", () => { ); }); - it("delivers stdin to the command via a nonce-delimited heredoc, still capped", () => { - const line = wrapCommand("n2", "base64 -d > '/workspace/a.txt'", Buffer.from("aGk="), 8); + it('delivers stdin to the command via a nonce-delimited heredoc, still capped', () => { + const line = wrapCommand('n2', "base64 -d > '/workspace/a.txt'", Buffer.from('aGk='), 8); expect(line).toBe( `printf '${SOH}B%s\\n' n2; { base64 -d > '/workspace/a.txt' <<'KAGENTI_EOF_n2'\n` + `aGk=\nKAGENTI_EOF_n2\n} | head -c 9 | base64; ` + @@ -37,75 +37,80 @@ describe("wrapCommand", () => { it("asks for cap+1 bytes so the client can distinguish 'exactly at the cap' from 'over it'", () => { // At exactly `cap` bytes the read is complete and must NOT be flagged; the client // detects truncation as `stdout.length > cap`, which needs one byte of evidence. - expect(wrapCommand("n3", "cat f", undefined, 100)).toContain("| head -c 101 |"); + expect(wrapCommand('n3', 'cat f', undefined, 100)).toContain('| head -c 101 |'); }); }); -describe("FrameParser", () => { +describe('FrameParser', () => { const SOHb = SOH; function frame(nonce: string, payload: string, code: number): string { - const b64 = Buffer.from(payload).toString("base64"); + const b64 = Buffer.from(payload).toString('base64'); return `${SOHb}B${nonce}\n${b64}\n${SOHb}E${nonce} ${code}\n`; } - it("emits a complete frame in one chunk, base64-decoded", () => { + it('emits a complete frame in one chunk, base64-decoded', () => { const p = new FrameParser(); - const frames = p.push(Buffer.from(frame("n1", "hello", 0))); + const frames = p.push(Buffer.from(frame('n1', 'hello', 0))); expect(frames).toHaveLength(1); - expect(frames[0]).toMatchObject({ nonce: "n1", exitCode: 0 }); - expect(frames[0].stdout.toString()).toBe("hello"); + expect(frames[0]).toMatchObject({ nonce: 'n1', exitCode: 0 }); + expect(frames[0].stdout.toString()).toBe('hello'); }); - it("waits for the end marker (no emit on a partial frame)", () => { + it('waits for the end marker (no emit on a partial frame)', () => { const p = new FrameParser(); - const full = frame("n1", "hello", 0); + const full = frame('n1', 'hello', 0); expect(p.push(Buffer.from(full.slice(0, 10)))).toHaveLength(0); const rest = p.push(Buffer.from(full.slice(10))); expect(rest).toHaveLength(1); - expect(rest[0].stdout.toString()).toBe("hello"); + expect(rest[0].stdout.toString()).toBe('hello'); }); - it("handles a split in the middle of the begin marker", () => { + it('handles a split in the middle of the begin marker', () => { const p = new FrameParser(); - const full = frame("n7", "x", 0); + const full = frame('n7', 'x', 0); const cut = 1; // mid "\x01B..." expect(p.push(Buffer.from(full.slice(0, cut)))).toHaveLength(0); expect(p.push(Buffer.from(full.slice(cut)))).toHaveLength(1); }); - it("emits multiple frames present in one chunk, preserving exit codes", () => { + it('emits multiple frames present in one chunk, preserving exit codes', () => { const p = new FrameParser(); - const frames = p.push(Buffer.from(frame("a", "one", 0) + frame("b", "two", 2))); + const frames = p.push(Buffer.from(frame('a', 'one', 0) + frame('b', 'two', 2))); expect(frames.map((f) => [f.nonce, f.stdout.toString(), f.exitCode])).toEqual([ - ["a", "one", 0], - ["b", "two", 2], + ['a', 'one', 0], + ['b', 'two', 2], ]); }); - it("round-trips binary payloads (NUL and high bytes)", () => { + it('round-trips binary payloads (NUL and high bytes)', () => { const p = new FrameParser(); const bin = Buffer.from([0x00, 0xff, 0x01, 0x42, 0x0a]); - const b64 = bin.toString("base64"); + const b64 = bin.toString('base64'); const frames = p.push(Buffer.from(`${SOHb}Bn1\n${b64}\n${SOHb}En1 0\n`)); expect(frames[0].stdout.equals(bin)).toBe(true); }); }); -describe("wrapCommand executed by a real bash (integration)", () => { - const hasBash = spawnSync("bash", ["-c", "true"]).status === 0; +describe('wrapCommand executed by a real bash (integration)', () => { + const hasBash = spawnSync('bash', ['-c', 'true']).status === 0; const maybe = hasBash ? it : it.skip; - maybe("writes a file through the heredoc-stdin path and frames exit 0", () => { - const dir = mkdtempSync(join(tmpdir(), "framing-bash-")); + maybe('writes a file through the heredoc-stdin path and frames exit 0', () => { + const dir = mkdtempSync(join(tmpdir(), 'framing-bash-')); try { - const target = join(dir, "out.txt"); + const target = join(dir, 'out.txt'); const content = Buffer.from('hello\nworld\nspecial "q" $x `b`\n'); - const line = wrapCommand("n1", `base64 -d > '${target}'`, Buffer.from(content.toString("base64")), 8 * 1024 * 1024); - const res = spawnSync("bash", { input: line }); + const line = wrapCommand( + 'n1', + `base64 -d > '${target}'`, + Buffer.from(content.toString('base64')), + 8 * 1024 * 1024, + ); + const res = spawnSync('bash', { input: line }); expect(res.status).toBe(0); const frames = new FrameParser().push(res.stdout); expect(frames).toHaveLength(1); - expect(frames[0]).toMatchObject({ nonce: "n1", exitCode: 0 }); + expect(frames[0]).toMatchObject({ nonce: 'n1', exitCode: 0 }); expect(existsSync(target)).toBe(true); expect(readFileSync(target).equals(content)).toBe(true); } finally { @@ -113,55 +118,63 @@ describe("wrapCommand executed by a real bash (integration)", () => { } }); - maybe("frames a non-zero exit code from a failing write", () => { - const line = wrapCommand("n2", "base64 -d > '/no_such_dir_xyz/out.txt'", Buffer.from("eA=="), 8 * 1024 * 1024); - const res = spawnSync("bash", { input: line }); + maybe('frames a non-zero exit code from a failing write', () => { + const line = wrapCommand( + 'n2', + "base64 -d > '/no_such_dir_xyz/out.txt'", + Buffer.from('eA=='), + 8 * 1024 * 1024, + ); + const res = spawnSync('bash', { input: line }); const frames = new FrameParser().push(res.stdout); expect(frames).toHaveLength(1); - expect(frames[0].nonce).toBe("n2"); + expect(frames[0].nonce).toBe('n2'); expect(frames[0].exitCode).not.toBe(0); }); - maybe("caps raw stdout at capBytes + 1 through a real pipeline", () => { + maybe('caps raw stdout at capBytes + 1 through a real pipeline', () => { // 5000 bytes offered, cap 100 → the pod hands back exactly 101, so the client sees // one byte past the cap and knows the output was cut. Proving this against a real // bash matters: the hermetic conformance fake simulates the cap, so it can only // confirm the client half of the contract. - const line = wrapCommand("n3", "yes AAAAAAAA | head -c 5000", undefined, 100); - const res = spawnSync("bash", { input: line }); + const line = wrapCommand('n3', 'yes AAAAAAAA | head -c 5000', undefined, 100); + const res = spawnSync('bash', { input: line }); const frames = new FrameParser().push(res.stdout); expect(frames).toHaveLength(1); expect(frames[0].stdout.length).toBe(101); }); - maybe("passes output through untouched when it lands under the cap", () => { - const line = wrapCommand("n4", "printf 'abc'", undefined, 100); - const res = spawnSync("bash", { input: line }); + maybe('passes output through untouched when it lands under the cap', () => { + const line = wrapCommand('n4', "printf 'abc'", undefined, 100); + const res = spawnSync('bash', { input: line }); const frames = new FrameParser().push(res.stdout); - expect(frames[0].stdout.toString()).toBe("abc"); + expect(frames[0].stdout.toString()).toBe('abc'); expect(frames[0].exitCode).toBe(0); }); - maybe("PIPESTATUS[0] still reports the command, not head or base64", () => { + maybe('PIPESTATUS[0] still reports the command, not head or base64', () => { // The cap adds a pipeline stage; if it were inserted before the group, or if the // printf read a different PIPESTATUS index, a failing command would frame as 0. - const line = wrapCommand("n5", "exit 42", undefined, 100); - const res = spawnSync("bash", { input: line }); + const line = wrapCommand('n5', 'exit 42', undefined, 100); + const res = spawnSync('bash', { input: line }); const frames = new FrameParser().push(res.stdout); expect(frames[0].exitCode).toBe(42); }); - maybe("reports CAP_STAGE_FAILED when `head` is missing, instead of empty-with-exit-0", () => { + maybe('reports CAP_STAGE_FAILED when `head` is missing, instead of empty-with-exit-0', () => { // THE data-loss scenario, reproduced rather than argued. With `head` absent the // pipeline yields empty stdout and the COMMAND's own exit 0 (verified: group=0, // head=127), so a read would come back as a successful empty buffer and Pi's Edit // would write that back over the file. Shadow `head` with a 127 stub on PATH and // assert the frame now carries the sentinel instead. - const dir = mkdtempSync(join(tmpdir(), "framing-nohead-")); + const dir = mkdtempSync(join(tmpdir(), 'framing-nohead-')); try { - writeFileSync(join(dir, "head"), "#!/bin/sh\nexit 127\n", { mode: 0o755 }); - const line = wrapCommand("n7", "printf abc", undefined, 100); - const res = spawnSync("bash", { input: line, env: { ...process.env, PATH: `${dir}:${process.env.PATH}` } }); + writeFileSync(join(dir, 'head'), '#!/bin/sh\nexit 127\n', { mode: 0o755 }); + const line = wrapCommand('n7', 'printf abc', undefined, 100); + const res = spawnSync('bash', { + input: line, + env: { ...process.env, PATH: `${dir}:${process.env.PATH}` }, + }); const frames = new FrameParser().push(res.stdout); expect(frames).toHaveLength(1); expect(frames[0].stdout.length).toBe(0); // the silent part: no output came back @@ -171,14 +184,21 @@ describe("wrapCommand executed by a real bash (integration)", () => { } }); - maybe("reports CAP_STAGE_FAILED when `head` exists but rejects -c (busybox-style)", () => { + maybe('reports CAP_STAGE_FAILED when `head` exists but rejects -c (busybox-style)', () => { // The other real variant, and it produces the identical silent signature (group=0, // head=1), so detecting only "command not found" would have missed it. - const dir = mkdtempSync(join(tmpdir(), "framing-badhead-")); + const dir = mkdtempSync(join(tmpdir(), 'framing-badhead-')); try { - writeFileSync(join(dir, "head"), '#!/bin/sh\necho "head: unrecognized option" >&2\nexit 1\n', { mode: 0o755 }); - const line = wrapCommand("n8", "printf abc", undefined, 100); - const res = spawnSync("bash", { input: line, env: { ...process.env, PATH: `${dir}:${process.env.PATH}` } }); + writeFileSync( + join(dir, 'head'), + '#!/bin/sh\necho "head: unrecognized option" >&2\nexit 1\n', + { mode: 0o755 }, + ); + const line = wrapCommand('n8', 'printf abc', undefined, 100); + const res = spawnSync('bash', { + input: line, + env: { ...process.env, PATH: `${dir}:${process.env.PATH}` }, + }); const frames = new FrameParser().push(res.stdout); expect(frames[0].exitCode).toBe(CAP_STAGE_FAILED); } finally { @@ -188,20 +208,25 @@ describe("wrapCommand executed by a real bash (integration)", () => { maybe("a healthy pipeline still reports the command's own status, not the sentinel", () => { // Guards the inverse: the stage check must not swallow real exit codes. - const line = wrapCommand("n9", "exit 42", undefined, 100); - const res = spawnSync("bash", { input: line }); + const line = wrapCommand('n9', 'exit 42', undefined, 100); + const res = spawnSync('bash', { input: line }); expect(new FrameParser().push(res.stdout)[0].exitCode).toBe(42); }); - maybe("still writes a file through the capped heredoc-stdin path", () => { + maybe('still writes a file through the capped heredoc-stdin path', () => { // A write produces no stdout, so head passes 0 bytes and the cap is inert — but the // heredoc must survive having a stage appended after the closing brace. - const dir = mkdtempSync(join(tmpdir(), "framing-bash-cap-")); + const dir = mkdtempSync(join(tmpdir(), 'framing-bash-cap-')); try { - const target = join(dir, "out.txt"); - const content = Buffer.from("payload\nwith newline\n"); - const line = wrapCommand("n6", `base64 -d > '${target}'`, Buffer.from(content.toString("base64")), 100); - const res = spawnSync("bash", { input: line }); + const target = join(dir, 'out.txt'); + const content = Buffer.from('payload\nwith newline\n'); + const line = wrapCommand( + 'n6', + `base64 -d > '${target}'`, + Buffer.from(content.toString('base64')), + 100, + ); + const res = spawnSync('bash', { input: line }); expect(res.status).toBe(0); const frames = new FrameParser().push(res.stdout); expect(frames[0].exitCode).toBe(0); diff --git a/packages/k8s-sandbox/test/grep-tool.test.ts b/packages/k8s-sandbox/test/grep-tool.test.ts index 7153652..b406889 100644 --- a/packages/k8s-sandbox/test/grep-tool.test.ts +++ b/packages/k8s-sandbox/test/grep-tool.test.ts @@ -1,14 +1,14 @@ -import { describe, expect, it } from "vitest"; -import type { ExecInPod } from "../src/exec.js"; -import type { K8sSandboxConfig } from "../src/config.js"; -import { createPodGrepTool } from "../src/grep-tool.js"; +import { describe, expect, it } from 'vitest'; +import type { ExecInPod } from '../src/exec.js'; +import type { K8sSandboxConfig } from '../src/config.js'; +import { createPodGrepTool } from '../src/grep-tool.js'; const cfg: K8sSandboxConfig = { - pod: "sbx-0", - namespace: "default", + pod: 'sbx-0', + namespace: 'default', context: undefined, - podCwd: "/workspace", - headCwd: "/head", + podCwd: '/workspace', + headCwd: '/head', }; /** Build a fake ExecInPod that returns a scripted result and records calls. */ @@ -17,41 +17,49 @@ function fakeExec(result: { stdout?: string; exitCode?: number | null; truncated const fn: ExecInPod = async (command) => { calls.push(command); const exitCode = result.exitCode === undefined ? 0 : result.exitCode; - return { stdout: Buffer.from(result.stdout ?? ""), exitCode, truncated: result.truncated ?? false }; + return { + stdout: Buffer.from(result.stdout ?? ''), + exitCode, + truncated: result.truncated ?? false, + }; }; return { fn, calls }; } -describe("createPodGrepTool", () => { - it("runs rg in the pod against the mapped path and returns match text", async () => { - const { fn, calls } = fakeExec({ stdout: "a.ts:1:hit" }); - const tool = createPodGrepTool("/head", fn, cfg); - const result = await tool.execute("t1", { pattern: "x", path: "/head" }); - expect(calls[0]).toContain("rg"); +describe('createPodGrepTool', () => { + it('runs rg in the pod against the mapped path and returns match text', async () => { + const { fn, calls } = fakeExec({ stdout: 'a.ts:1:hit' }); + const tool = createPodGrepTool('/head', fn, cfg); + const result = await tool.execute('t1', { pattern: 'x', path: '/head' }); + expect(calls[0]).toContain('rg'); expect(calls[0]).toContain("'/workspace'"); - expect((result as { content: Array<{ text: string }> }).content[0].text).toBe("a.ts:1:hit"); + expect((result as { content: Array<{ text: string }> }).content[0].text).toBe('a.ts:1:hit'); }); - it("reports a cap trip as truncation, not as an rg failure", async () => { + it('reports a cap trip as truncation, not as an rg failure', async () => { // Today truncation lands in the `exitCode !== 0 && !== 1` branch and surfaces as // "rg failed in pod (exit null)", which blames ripgrep for our own cap. - const { fn } = fakeExec({ stdout: "a.ts:1:hit", exitCode: null, truncated: true }); - const tool = createPodGrepTool("/head", fn, cfg); - await expect(tool.execute("t1", { pattern: "x", path: "/head" })).rejects.toThrow(/output cap/); + const { fn } = fakeExec({ stdout: 'a.ts:1:hit', exitCode: null, truncated: true }); + const tool = createPodGrepTool('/head', fn, cfg); + await expect(tool.execute('t1', { pattern: 'x', path: '/head' })).rejects.toThrow(/output cap/); }); - it("still reports a genuine rg failure distinctly from a cap trip", async () => { - const { fn } = fakeExec({ stdout: "", exitCode: 2, truncated: false }); - const tool = createPodGrepTool("/head", fn, cfg); - const err = (await tool.execute("t1", { pattern: "[", path: "/head" }).catch((e) => e)) as Error; + it('still reports a genuine rg failure distinctly from a cap trip', async () => { + const { fn } = fakeExec({ stdout: '', exitCode: 2, truncated: false }); + const tool = createPodGrepTool('/head', fn, cfg); + const err = (await tool + .execute('t1', { pattern: '[', path: '/head' }) + .catch((e) => e)) as Error; expect(err.message).toMatch(/rg failed in pod/); expect(err.message).not.toMatch(/output cap/); }); it("returns 'No matches found' on exit 1 with empty output", async () => { - const { fn } = fakeExec({ stdout: "", exitCode: 1, truncated: false }); - const tool = createPodGrepTool("/head", fn, cfg); - const result = await tool.execute("t1", { pattern: "nope", path: "/head" }); - expect((result as { content: Array<{ text: string }> }).content[0].text).toBe("No matches found"); + const { fn } = fakeExec({ stdout: '', exitCode: 1, truncated: false }); + const tool = createPodGrepTool('/head', fn, cfg); + const result = await tool.execute('t1', { pattern: 'nope', path: '/head' }); + expect((result as { content: Array<{ text: string }> }).content[0].text).toBe( + 'No matches found', + ); }); }); diff --git a/packages/k8s-sandbox/test/grpc-relay-transport.test.ts b/packages/k8s-sandbox/test/grpc-relay-transport.test.ts index 4ceb7cb..e2ef932 100644 --- a/packages/k8s-sandbox/test/grpc-relay-transport.test.ts +++ b/packages/k8s-sandbox/test/grpc-relay-transport.test.ts @@ -1,16 +1,13 @@ -import { EventEmitter } from "node:events"; -import { describe, expect, it, vi } from "vitest"; -import { runConformance, type FakeBehavior, type TransportFactory } from "./conformance.js"; -import { - GrpcRelayTransport, - type ExecClientLike, -} from "../src/grpc-relay-transport.js"; +import { EventEmitter } from 'node:events'; +import { describe, expect, it, vi } from 'vitest'; +import { runConformance, type FakeBehavior, type TransportFactory } from './conformance.js'; +import { GrpcRelayTransport, type ExecClientLike } from '../src/grpc-relay-transport.js'; import { Stream, type AbortRequest, type ExecEvent, type ExecRequest, -} from "../src/gen/sandbox/v1/sandbox.js"; +} from '../src/gen/sandbox/v1/sandbox.js'; /** Build a fake ExecClientLike that scripts a FakeBehavior onto ExecEvent frames. */ function fakeClient(behavior: FakeBehavior): { @@ -33,11 +30,15 @@ function fakeClient(behavior: FakeBehavior): { queueMicrotask(() => { if (behavior.hang) return; // never completes for (const s of behavior.stdout ?? []) - stream.emit("data", { chunk: { reqId, data: Buffer.from(s), stream: Stream.STREAM_STDOUT } } as ExecEvent); + stream.emit('data', { + chunk: { reqId, data: Buffer.from(s), stream: Stream.STREAM_STDOUT }, + } as ExecEvent); for (const s of behavior.stderr ?? []) - stream.emit("data", { chunk: { reqId, data: Buffer.from(s), stream: Stream.STREAM_STDERR } } as ExecEvent); - stream.emit("data", { end: { reqId, exitCode: behavior.exitCode ?? 0 } } as ExecEvent); - stream.emit("end"); + stream.emit('data', { + chunk: { reqId, data: Buffer.from(s), stream: Stream.STREAM_STDERR }, + } as ExecEvent); + stream.emit('data', { end: { reqId, exitCode: behavior.exitCode ?? 0 } } as ExecEvent); + stream.emit('end'); }); return stream; }, @@ -52,7 +53,7 @@ function fakeClient(behavior: FakeBehavior): { const grpcFactory: TransportFactory = (behavior, opts) => { const { client, stdinSeen, aborted, reqIdSeen } = fakeClient(behavior); - const transport = GrpcRelayTransport("sbx-1", client, { outputCapBytes: opts?.outputCapBytes }); + const transport = GrpcRelayTransport('sbx-1', client, { outputCapBytes: opts?.outputCapBytes }); return { transport, stdinSeen, @@ -61,12 +62,12 @@ const grpcFactory: TransportFactory = (behavior, opts) => { // flood running. Correlate rather than just counting calls. producerStop: () => { const id = reqIdSeen(); - return id !== undefined && aborted().includes(id) ? "remote-abort" : "none"; + return id !== undefined && aborted().includes(id) ? 'remote-abort' : 'none'; }, }; }; -runConformance("GrpcRelayTransport", grpcFactory, { producerStop: "remote-abort", streams: true }); +runConformance('GrpcRelayTransport', grpcFactory, { producerStop: 'remote-abort', streams: true }); function manualClient() { let stream!: EventEmitter & { cancel: () => void }; @@ -92,75 +93,77 @@ function manualClient() { }; return { client, - emit: (ev: ExecEvent) => stream.emit("data", ev), + emit: (ev: ExecEvent) => stream.emit('data', ev), aborted: () => aborted, reqId: () => lastReqId, }; } -describe("GrpcRelayTransport extra semantics", () => { +describe('GrpcRelayTransport extra semantics', () => { it("the cap's Abort names the exec's own req_id", async () => { // The shared battery only asks "was the producer stopped". Here we pin the wire // detail the battery cannot express portably: the relay routes an Abort by req_id, // so an Abort carrying any other id would leave the flood running at full rate // while this exec has already stopped reading (spec §8). const { client, emit, aborted, reqId } = manualClient(); - const t = GrpcRelayTransport("sbx-1", client as never, { outputCapBytes: 6 }); - const p = t.exec("cat big"); + const t = GrpcRelayTransport('sbx-1', client as never, { outputCapBytes: 6 }); + const p = t.exec('cat big'); await Promise.resolve(); // let exec() register its handlers emit({ - chunk: { reqId: reqId()!, data: Buffer.from("aaaabbbb"), stream: Stream.STREAM_STDOUT }, + chunk: { reqId: reqId()!, data: Buffer.from('aaaabbbb'), stream: Stream.STREAM_STDOUT }, } as ExecEvent); const r = await p; - expect(r.stdout.toString()).toContain("[output truncated]"); + expect(r.stdout.toString()).toContain('[output truncated]'); expect(aborted()).toContain(reqId()); }); - it("dedups: a late End for a settled reqId is dropped", async () => { + it('dedups: a late End for a settled reqId is dropped', async () => { const { client, emit } = manualClient(); - const t = GrpcRelayTransport("sbx-1", client as never); - const p = t.exec("echo hi"); - emit({ chunk: { reqId: 1, data: Buffer.from("hi"), stream: Stream.STREAM_STDOUT } } as ExecEvent); + const t = GrpcRelayTransport('sbx-1', client as never); + const p = t.exec('echo hi'); + emit({ + chunk: { reqId: 1, data: Buffer.from('hi'), stream: Stream.STREAM_STDOUT }, + } as ExecEvent); emit({ end: { reqId: 1, exitCode: 0 } } as ExecEvent); const r = await p; - expect(r.stdout.toString()).toBe("hi"); + expect(r.stdout.toString()).toBe('hi'); // A duplicate terminal frame after settlement must not throw or change the result. expect(() => emit({ end: { reqId: 1, exitCode: 9 } } as ExecEvent)).not.toThrow(); }); - it("harness deadline fires independently of worker timeout_s", async () => { + it('harness deadline fires independently of worker timeout_s', async () => { vi.useFakeTimers(); const { client } = manualClient(); - const t = GrpcRelayTransport("sbx-1", client as never, { deadlineMs: 500 }); - const p = t.exec("sleep 999"); // no exec opts.timeout ⇒ deadlineMs governs + const t = GrpcRelayTransport('sbx-1', client as never, { deadlineMs: 500 }); + const p = t.exec('sleep 999'); // no exec opts.timeout ⇒ deadlineMs governs const assertion = expect(p).rejects.toThrow(/^timeout:/); await vi.advanceTimersByTimeAsync(500); await assertion; vi.useRealTimers(); }); - it("close() closes the underlying gRPC channel via client.close()", async () => { + it('close() closes the underlying gRPC channel via client.close()', async () => { const { client } = manualClient(); const closeSpy = vi.fn(); (client as unknown as { close: () => void }).close = closeSpy; - const t = GrpcRelayTransport("sbx-1", client as never); + const t = GrpcRelayTransport('sbx-1', client as never); await t.close(); expect(closeSpy).toHaveBeenCalledTimes(1); }); - it("close() is idempotent: calling it twice closes the channel at most once", async () => { + it('close() is idempotent: calling it twice closes the channel at most once', async () => { const { client } = manualClient(); const closeSpy = vi.fn(); (client as unknown as { close: () => void }).close = closeSpy; - const t = GrpcRelayTransport("sbx-1", client as never); + const t = GrpcRelayTransport('sbx-1', client as never); await t.close(); await t.close(); expect(closeSpy).toHaveBeenCalledTimes(1); }); - it("close() is a safe no-op when the client exposes no close() (scripted fakes)", async () => { + it('close() is a safe no-op when the client exposes no close() (scripted fakes)', async () => { const { client } = manualClient(); // manualClient's fake has no `close` method - const t = GrpcRelayTransport("sbx-1", client as never); + const t = GrpcRelayTransport('sbx-1', client as never); await expect(t.close()).resolves.toBeUndefined(); }); }); diff --git a/packages/k8s-sandbox/test/live-relay.test.ts b/packages/k8s-sandbox/test/live-relay.test.ts index f2835c8..4b7e955 100644 --- a/packages/k8s-sandbox/test/live-relay.test.ts +++ b/packages/k8s-sandbox/test/live-relay.test.ts @@ -1,13 +1,13 @@ -import { type ChildProcess, execFileSync, spawn } from "node:child_process"; -import { mkdtempSync, rmSync } from "node:fs"; -import { tmpdir } from "node:os"; -import path from "node:path"; -import { fileURLToPath } from "node:url"; -import { credentials } from "@grpc/grpc-js"; -import { afterAll, beforeAll, describe, expect, it } from "vitest"; -import { GrpcRelayTransport, type ExecClientLike } from "../src/grpc-relay-transport.js"; -import { OUTPUT_TRUNCATED_MARKER } from "../src/transport.js"; -import { SandboxExecClient } from "../src/gen/sandbox/v1/sandbox.js"; +import { type ChildProcess, execFileSync, spawn } from 'node:child_process'; +import { mkdtempSync, rmSync } from 'node:fs'; +import { tmpdir } from 'node:os'; +import path from 'node:path'; +import { fileURLToPath } from 'node:url'; +import { credentials } from '@grpc/grpc-js'; +import { afterAll, beforeAll, describe, expect, it } from 'vitest'; +import { GrpcRelayTransport, type ExecClientLike } from '../src/grpc-relay-transport.js'; +import { OUTPUT_TRUNCATED_MARKER } from '../src/transport.js'; +import { SandboxExecClient } from '../src/gen/sandbox/v1/sandbox.js'; /** * Live counterpart to the hermetic conformance battery. The battery's fakes are @@ -30,10 +30,10 @@ import { SandboxExecClient } from "../src/gen/sandbox/v1/sandbox.js"; * (worker-disconnect) spawns and kills its OWN worker process under a distinct * sandbox id, so the test is self-contained and repeatable — see makeSelfManagedWorker. */ -const LIVE = process.env.SH_LIVE_RELAY === "1"; -const SANDBOX_ID = process.env.SANDBOX_ID ?? "sbx-dev-1"; -const RELAY_ADDR = process.env.SH_RELAY_ADDR ?? "localhost:8443"; -const SANDBOX_TOKEN = process.env.SANDBOX_TOKEN ?? "dev-token"; +const LIVE = process.env.SH_LIVE_RELAY === '1'; +const SANDBOX_ID = process.env.SANDBOX_ID ?? 'sbx-dev-1'; +const RELAY_ADDR = process.env.SH_RELAY_ADDR ?? 'localhost:8443'; +const SANDBOX_TOKEN = process.env.SANDBOX_TOKEN ?? 'dev-token'; /** Same two-line construction as select-sandbox.ts:52's defaultExecClient — one dialing idiom in the repo. */ function makeExecClient(addr: string): ExecClientLike { @@ -55,13 +55,16 @@ function makeLiveTransport(opts: { outputCapBytes?: number } = {}, sandboxId: st let workerBinDir: string | undefined; let workerBinPath: string | undefined; -const remoteWorkerDir = fileURLToPath(new URL("../../../remote-worker", import.meta.url)); +const remoteWorkerDir = fileURLToPath(new URL('../../../remote-worker', import.meta.url)); beforeAll(() => { if (!LIVE) return; - workerBinDir = mkdtempSync(path.join(tmpdir(), "sh-live-relay-worker-")); - workerBinPath = path.join(workerBinDir, "worker"); - execFileSync("go", ["build", "-o", workerBinPath, "./cmd/worker"], { cwd: remoteWorkerDir, stdio: "pipe" }); + workerBinDir = mkdtempSync(path.join(tmpdir(), 'sh-live-relay-worker-')); + workerBinPath = path.join(workerBinDir, 'worker'); + execFileSync('go', ['build', '-o', workerBinPath, './cmd/worker'], { + cwd: remoteWorkerDir, + stdio: 'pipe', + }); }); afterAll(() => { @@ -71,7 +74,7 @@ afterAll(() => { /** Waits for a line matching `pattern` on the child's stdout or stderr (Go's `log` writes to stderr). */ function waitForLine(child: ChildProcess, pattern: RegExp, timeoutMs: number): Promise { return new Promise((resolve, reject) => { - let buf = ""; + let buf = ''; const onData = (d: Buffer) => { buf += d.toString(); if (pattern.test(buf)) { @@ -89,19 +92,19 @@ function waitForLine(child: ChildProcess, pattern: RegExp, timeoutMs: number): P }, timeoutMs); const cleanup = () => { clearTimeout(timer); - child.stdout?.removeListener("data", onData); - child.stderr?.removeListener("data", onData); - child.removeListener("exit", onExit); + child.stdout?.removeListener('data', onData); + child.stderr?.removeListener('data', onData); + child.removeListener('exit', onExit); }; - child.stdout?.on("data", onData); - child.stderr?.on("data", onData); - child.on("exit", onExit); + child.stdout?.on('data', onData); + child.stderr?.on('data', onData); + child.on('exit', onExit); }); } /** Spawns the compiled worker binary attached under `sandboxId` and waits for it to attach to the relay. */ async function spawnSelfManagedWorker(sandboxId: string): Promise { - if (!workerBinPath) throw new Error("worker binary not built — beforeAll did not run"); + if (!workerBinPath) throw new Error('worker binary not built — beforeAll did not run'); const child = spawn(workerBinPath, [], { env: { ...process.env, @@ -113,14 +116,14 @@ async function spawnSelfManagedWorker(sandboxId: string): Promise try { await waitForLine(child, /attached, serving execs/, 10_000); } catch (err) { - child.kill("SIGKILL"); + child.kill('SIGKILL'); throw err; } return child; } -describe.skipIf(!LIVE)("GrpcRelayTransport against a live relay + worker", () => { - it("enforces the dual-ended timeout against a real long-running command", async () => { +describe.skipIf(!LIVE)('GrpcRelayTransport against a live relay + worker', () => { + it('enforces the dual-ended timeout against a real long-running command', async () => { const t = makeLiveTransport(); // The worker kills its own child at timeout_s AND the harness has its own deadline; // whichever fires, the caller must see timeout:2 rather than hang (spec §8). Asserting @@ -129,19 +132,19 @@ describe.skipIf(!LIVE)("GrpcRelayTransport against a live relay + worker", () => // via the transport's own 120s default deadline, but it would blow past this bound. const started = Date.now(); try { - await expect(t.exec("sleep 30", { timeout: 2 })).rejects.toThrow("timeout:2"); + await expect(t.exec('sleep 30', { timeout: 2 })).rejects.toThrow('timeout:2'); expect(Date.now() - started).toBeLessThan(10_000); } finally { await t.close(); } }, 30_000); - it("truncates a real flood at the cap and marks it", async () => { + it('truncates a real flood at the cap and marks it', async () => { const t = makeLiveTransport({ outputCapBytes: 64 * 1024 }); // yes | head -c is a genuine multi-chunk flood through real 32 KiB Chunk frames, // which is the path the hermetic test's scripted frames only imitate. try { - const r = await t.exec("yes AAAAAAAA | head -c 1000000"); + const r = await t.exec('yes AAAAAAAA | head -c 1000000'); expect(r.stdout.toString()).toContain(OUTPUT_TRUNCATED_MARKER); // Near the 64 KiB cap, not the full ~1MB the command produced. expect(r.stdout.length).toBeLessThan(200 * 1024); @@ -150,7 +153,7 @@ describe.skipIf(!LIVE)("GrpcRelayTransport against a live relay + worker", () => } }, 30_000); - it("fails an in-flight exec when the worker disconnects, rather than hanging", async () => { + it('fails an in-flight exec when the worker disconnects, rather than hanging', async () => { // relay.ts's Attach teardown pushes {error: "worker disconnected"} into every live // sink. That is what lets run-leaf retry onto a healthy sandbox instead of blocking // on a dead one (§10: no mid-exec durability). @@ -162,17 +165,17 @@ describe.skipIf(!LIVE)("GrpcRelayTransport against a live relay + worker", () => const worker = await spawnSelfManagedWorker(disconnectId); try { const t = makeLiveTransport({}, disconnectId); - const p = t.exec("sleep 20"); + const p = t.exec('sleep 20'); // Let the exec actually land: the relay must have parked the sink and written // ServerFrame{exec} to the worker before we kill it, or we'd just be testing // "Attach never happened," not "Attach was torn down mid-exec". await new Promise((r) => setTimeout(r, 1000)); - worker.kill("SIGKILL"); + worker.kill('SIGKILL'); await expect(p).rejects.toThrow(/worker disconnected|timeout|CANCELLED|UNAVAILABLE/); await t.close(); } finally { // Idempotent safety net: a process already dead ignores a second SIGKILL. - worker.kill("SIGKILL"); + worker.kill('SIGKILL'); } }, 60_000); }); diff --git a/packages/k8s-sandbox/test/m3-live-smoke.test.ts b/packages/k8s-sandbox/test/m3-live-smoke.test.ts index a9e16f4..c708996 100644 --- a/packages/k8s-sandbox/test/m3-live-smoke.test.ts +++ b/packages/k8s-sandbox/test/m3-live-smoke.test.ts @@ -1,17 +1,17 @@ -import { spawn as nodeSpawn } from "node:child_process"; -import { execFile } from "node:child_process"; -import { promisify } from "node:util"; -import { beforeAll, describe, expect, it } from "vitest"; -import type { K8sSandboxConfig } from "../src/config.js"; -import { KubectlTransport } from "../src/exec.js"; -import { persistentExecInPod } from "../src/persistent-exec.js"; +import { spawn as nodeSpawn } from 'node:child_process'; +import { execFile } from 'node:child_process'; +import { promisify } from 'node:util'; +import { beforeAll, describe, expect, it } from 'vitest'; +import type { K8sSandboxConfig } from '../src/config.js'; +import { KubectlTransport } from '../src/exec.js'; +import { persistentExecInPod } from '../src/persistent-exec.js'; import { createPodBashOps, createPodFindOps, createPodLsOps, createPodReadOps, createPodWriteOps, -} from "../src/operations.js"; +} from '../src/operations.js'; const execFileP = promisify(execFile); @@ -22,23 +22,23 @@ const LIVE = !!process.env.M3_LIVE_SMOKE; // Construct the config directly (mirrors the unit-test fixture) so path mapping // is deterministic: head path /head/X maps to pod path /workspace/X. const cfg: K8sSandboxConfig = { - pod: process.env.KAGENTI_SANDBOX_POD ?? "", - namespace: process.env.KAGENTI_SANDBOX_NAMESPACE ?? "default", - context: process.env.KAGENTI_SANDBOX_CONTEXT ?? "kind-kagenti", - podCwd: "/workspace", - headCwd: "/head", + pod: process.env.KAGENTI_SANDBOX_POD ?? '', + namespace: process.env.KAGENTI_SANDBOX_NAMESPACE ?? 'default', + context: process.env.KAGENTI_SANDBOX_CONTEXT ?? 'kind-kagenti', + podCwd: '/workspace', + headCwd: '/head', }; /** Direct `kubectl exec` into the pod (independent verification path). */ async function kubectlExecRaw(args: string[]): Promise { - const base = ["exec", "-n", cfg.namespace]; - if (cfg.context) base.push("--context", cfg.context); - base.push(cfg.pod, "--", ...args); - const { stdout } = await execFileP("kubectl", base, { maxBuffer: 8 * 1024 * 1024 }); + const base = ['exec', '-n', cfg.namespace]; + if (cfg.context) base.push('--context', cfg.context); + base.push(cfg.pod, '--', ...args); + const { stdout } = await execFileP('kubectl', base, { maxBuffer: 8 * 1024 * 1024 }); return stdout; } -describe.skipIf(!LIVE)("M3 live smoke (real kind cluster)", () => { +describe.skipIf(!LIVE)('M3 live smoke (real kind cluster)', () => { // Seed the find fixtures this suite reads. Claims 1, 4 and 5 assert on a `.gitignore` // and a tree of seeded `.ts` files that they do NOT create themselves — Claims 2 and 4b // build their own, which is why they pass on a bare pod while the others do not. @@ -54,16 +54,16 @@ describe.skipIf(!LIVE)("M3 live smoke (real kind cluster)", () => { // as an arbitrary non-root UID and `/tmp` is not writable. beforeAll(async () => { await kubectlExecRaw([ - "bash", - "-c", - "cd /workspace && mkdir -p src node_modules/pkg .git dist && " + + 'bash', + '-c', + 'cd /workspace && mkdir -p src node_modules/pkg .git dist && ' + 'printf "node_modules/\ndist/\n" > .gitignore && ' + - ": > src/keep.ts && : > node_modules/pkg/skip.ts && : > .git/cfg.ts && " + - ": > dist/bundle.ts && : > top.ts", + ': > src/keep.ts && : > node_modules/pkg/skip.ts && : > .git/cfg.ts && ' + + ': > dist/bundle.ts && : > top.ts', ]); }, 60_000); - it("Claim 1: a single persistent process serves a burst of >=3 ops", async () => { + it('Claim 1: a single persistent process serves a burst of >=3 ops', async () => { let spawnCount = 0; const countingSpawn = ((...a: Parameters) => { spawnCount += 1; @@ -83,13 +83,13 @@ describe.skipIf(!LIVE)("M3 live smoke (real kind cluster)", () => { // NOTE: the seeded .ts files are 0 bytes, so we assert presence of the // *.gitignore* read (non-empty) and structural results for ls/glob; // content length of empty files is intentionally not asserted. - const gi = await read.readFile("/head/.gitignore"); - const listing = await ls.readdir("/head"); - const globbed = await find.glob("*.ts", "/head", { - ignore: ["**/node_modules/**", "**/.git/**"], + const gi = await read.readFile('/head/.gitignore'); + const listing = await ls.readdir('/head'); + const globbed = await find.glob('*.ts', '/head', { + ignore: ['**/node_modules/**', '**/.git/**'], limit: 100, }); - const gi2 = await read.readFile("/head/.gitignore"); + const gi2 = await read.readFile('/head/.gitignore'); expect(gi.length).toBeGreaterThan(0); expect(listing.length).toBeGreaterThan(0); @@ -106,12 +106,11 @@ describe.skipIf(!LIVE)("M3 live smoke (real kind cluster)", () => { } }, 30000); - it("Claim 2 (TOP): write/edit over the persistent channel round-trips multi-line/special content", async () => { + it('Claim 2 (TOP): write/edit over the persistent channel round-trips multi-line/special content', async () => { const fastExec = persistentExecInPod(cfg, { fallback: KubectlTransport(cfg).exec }); - const headPath = "/head/m3-write.txt"; - const podPath = "/workspace/m3-write.txt"; - const content = - 'line1\nline2 with "quotes" and $dollar and `backtick`\nline3 end\n'; + const headPath = '/head/m3-write.txt'; + const podPath = '/workspace/m3-write.txt'; + const content = 'line1\nline2 with "quotes" and $dollar and `backtick`\nline3 end\n'; try { const write = createPodWriteOps(fastExec.exec, cfg); const read = createPodReadOps(fastExec.exec, cfg); @@ -125,7 +124,7 @@ describe.skipIf(!LIVE)("M3 live smoke (real kind cluster)", () => { expect(roundTrip).toBe(content); // Independently confirm with a direct kubectl exec cat. - const direct = await kubectlExecRaw(["cat", podPath]); + const direct = await kubectlExecRaw(['cat', podPath]); expect(direct).toBe(content); // eslint-disable-next-line no-console @@ -135,18 +134,18 @@ describe.skipIf(!LIVE)("M3 live smoke (real kind cluster)", () => { } }, 30000); - it("Claim 3: env injection reaches the bash op", async () => { + it('Claim 3: env injection reaches the bash op', async () => { const streamExec = KubectlTransport(cfg).exec; const bash = createPodBashOps(streamExec, cfg); const chunks: Buffer[] = []; - const r = await bash.exec("echo MARKER=$M3_SMOKE", "/head", { + const r = await bash.exec('echo MARKER=$M3_SMOKE', '/head', { onData: (d) => chunks.push(d), - env: { M3_SMOKE: "works-42" }, + env: { M3_SMOKE: 'works-42' }, }); const out = Buffer.concat(chunks).toString(); expect(r.exitCode).toBe(0); - expect(out).toContain("MARKER=works-42"); - const line = out.split("\n").find((l) => l.includes("MARKER=")) ?? out.trim(); + expect(out).toContain('MARKER=works-42'); + const line = out.split('\n').find((l) => l.includes('MARKER=')) ?? out.trim(); // eslint-disable-next-line no-console console.log(`[Claim3] captured: ${line.trim()}`); }, 30000); @@ -155,8 +154,8 @@ describe.skipIf(!LIVE)("M3 live smoke (real kind cluster)", () => { const fastExec = persistentExecInPod(cfg, { fallback: KubectlTransport(cfg).exec }); try { const find = createPodFindOps(fastExec.exec, cfg); - const results = await find.glob("*.ts", "/head", { - ignore: ["**/node_modules/**", "**/.git/**"], + const results = await find.glob('*.ts', '/head', { + ignore: ['**/node_modules/**', '**/.git/**'], limit: 100, }); const set = new Set(results); @@ -164,11 +163,11 @@ describe.skipIf(!LIVE)("M3 live smoke (real kind cluster)", () => { console.log(`[Claim4] glob result: ${JSON.stringify(results)}`); // Included: regular tracked files. - expect(set.has("src/keep.ts")).toBe(true); - expect(set.has("top.ts")).toBe(true); + expect(set.has('src/keep.ts')).toBe(true); + expect(set.has('top.ts')).toBe(true); // Excluded: in the ignore-list / rg built-ins. - expect(set.has("node_modules/pkg/skip.ts")).toBe(false); - expect(set.has(".git/cfg.ts")).toBe(false); + expect(set.has('node_modules/pkg/skip.ts')).toBe(false); + expect(set.has('.git/cfg.ts')).toBe(false); // ── gitignored DIRECTORY case (the nuance) ──────────────────────────── // operations.ts notes that a positive `-g ` is a ripgrep @@ -180,7 +179,7 @@ describe.skipIf(!LIVE)("M3 live smoke (real kind cluster)", () => { // though `*.ts` matches it. (Only `rg --files -uu`/`--no-ignore-vcs` // would surface it.) Assert the dir-prune behaviour here; the file-level // override is asserted separately in Claim 4b. - const distVisible = set.has("dist/bundle.ts"); + const distVisible = set.has('dist/bundle.ts'); // eslint-disable-next-line no-console console.log( `[Claim4] dist/bundle.ts visible via -g glob = ${distVisible} ` + @@ -192,7 +191,7 @@ describe.skipIf(!LIVE)("M3 live smoke (real kind cluster)", () => { } }, 30000); - it("Claim 4b: positive -g whitelist-overrides a file-level .gitignore (verified nuance)", async () => { + it('Claim 4b: positive -g whitelist-overrides a file-level .gitignore (verified nuance)', async () => { const fastExec = persistentExecInPod(cfg, { fallback: KubectlTransport(cfg).exec }); try { // Seed an ISOLATED fixture (does not touch the shared /workspace files): @@ -200,18 +199,18 @@ describe.skipIf(!LIVE)("M3 live smoke (real kind cluster)", () => { // the file `a.ts` by name. A positive `-g '*.ts'` should whitelist-override // that FILE-level ignore (unlike the DIRECTORY-prune case in Claim 4). await kubectlExecRaw([ - "bash", - "-lc", - "mkdir -p /workspace/ovr && " + - ": > /workspace/ovr/a.ts && " + - ": > /workspace/ovr/keep2.ts && " + + 'bash', + '-lc', + 'mkdir -p /workspace/ovr && ' + + ': > /workspace/ovr/a.ts && ' + + ': > /workspace/ovr/keep2.ts && ' + "printf 'a.ts\\n' > /workspace/ovr/.gitignore", ]); const find = createPodFindOps(fastExec.exec, cfg); // cwd /head/ovr maps to /workspace/ovr; empty ignore-list so ONLY // .gitignore is in play. - const results = await find.glob("*.ts", "/head/ovr", { + const results = await find.glob('*.ts', '/head/ovr', { ignore: [], limit: 100, }); @@ -221,25 +220,25 @@ describe.skipIf(!LIVE)("M3 live smoke (real kind cluster)", () => { // a.ts is gitignored by name, but the positive -g '*.ts' whitelist // overrides a FILE-level ignore -> it reappears. - expect(set.has("a.ts")).toBe(true); + expect(set.has('a.ts')).toBe(true); // keep2.ts is not ignored at all. - expect(set.has("keep2.ts")).toBe(true); + expect(set.has('keep2.ts')).toBe(true); } finally { - await kubectlExecRaw(["rm", "-rf", "/workspace/ovr"]); + await kubectlExecRaw(['rm', '-rf', '/workspace/ovr']); await fastExec.close(); } }, 30000); - it("Claim 5: close is non-throwing (best-effort process-count probe)", async () => { + it('Claim 5: close is non-throwing (best-effort process-count probe)', async () => { const fastExec = persistentExecInPod(cfg, { fallback: KubectlTransport(cfg).exec }); // Warm the channel so a persistent bash exists in the pod. - await createPodReadOps(fastExec.exec, cfg).readFile("/head/.gitignore"); + await createPodReadOps(fastExec.exec, cfg).readFile('/head/.gitignore'); const countBash = async (): Promise => { try { const out = await kubectlExecRaw([ - "sh", - "-c", + 'sh', + '-c', 'ps -o pid,args 2>/dev/null | grep -c "[b]ash"', ]); return parseInt(out.trim(), 10) || 0; @@ -258,19 +257,19 @@ describe.skipIf(!LIVE)("M3 live smoke (real kind cluster)", () => { console.log(`[Claim5] bash-process count before=${before} after=${after} (informational)`); }, 30000); - it("Claim 6: a file over the output cap fails the read loudly, with a usable message", async () => { + it('Claim 6: a file over the output cap fails the read loudly, with a usable message', async () => { // The one change with no hermetic proxy: a real `head -c` in a real pipeline, over // kubectl exec, against the sandbox image's coreutils. The unit tests prove the // command shape and the client half; only this proves they compose in the pod. const t = persistentExecInPod(cfg, { fallback: KubectlTransport(cfg).exec }); try { - const big = "/workspace/cap-probe.bin"; + const big = '/workspace/cap-probe.bin'; // 9 MiB > the 8 MiB cap, written in-pod so no large payload crosses the wire. - await kubectlExecRaw(["bash", "-c", `head -c 9437184 /dev/zero > ${big}`]); + await kubectlExecRaw(['bash', '-c', `head -c 9437184 /dev/zero > ${big}`]); const read = createPodReadOps(t.exec, cfg); // One read, three assertions: re-invoking would push 9 MiB through the pod's // pipeline three times for no extra coverage. - const err = (await read.readFile("/head/cap-probe.bin").catch((e) => e)) as Error; + const err = (await read.readFile('/head/cap-probe.bin').catch((e) => e)) as Error; expect(err.message).toMatch(/exceeds the .* output cap/); // The message must carry the size and the escape hatch, or the model cannot act. expect(err.message).toMatch(/9437184/); @@ -278,20 +277,22 @@ describe.skipIf(!LIVE)("M3 live smoke (real kind cluster)", () => { // A file just under the cap must still read cleanly — the cap must not have made // the whole path fragile. - const small = "/workspace/cap-probe-small.bin"; - await kubectlExecRaw(["bash", "-c", `head -c 1048576 /dev/zero > ${small}`]); - const buf = await read.readFile("/head/cap-probe-small.bin"); + const small = '/workspace/cap-probe-small.bin'; + await kubectlExecRaw(['bash', '-c', `head -c 1048576 /dev/zero > ${small}`]); + const buf = await read.readFile('/head/cap-probe-small.bin'); expect(buf.length).toBe(1048576); // And bash reports the kill rather than success (#181), over the same real pod. const bash = createPodBashOps(KubectlTransport(cfg).exec, cfg); - const r = await bash.exec(`cat ${big}`, "/head", { onData: () => {} }); + const r = await bash.exec(`cat ${big}`, '/head', { onData: () => {} }); expect(r.exitCode).toBe(137); } finally { await t.close(); - await kubectlExecRaw(["bash", "-c", "rm -f /workspace/cap-probe.bin /workspace/cap-probe-small.bin"]).catch( - () => {}, - ); + await kubectlExecRaw([ + 'bash', + '-c', + 'rm -f /workspace/cap-probe.bin /workspace/cap-probe-small.bin', + ]).catch(() => {}); } }, 120_000); }); diff --git a/packages/k8s-sandbox/test/operations.test.ts b/packages/k8s-sandbox/test/operations.test.ts index eaed9dc..ef9579f 100644 --- a/packages/k8s-sandbox/test/operations.test.ts +++ b/packages/k8s-sandbox/test/operations.test.ts @@ -1,7 +1,7 @@ -import { describe, expect, it, vi } from "vitest"; -import type { ExecInPod } from "../src/exec.js"; -import type { K8sSandboxConfig } from "../src/config.js"; -import { OUTPUT_TRUNCATED_MARKER } from "../src/transport.js"; +import { describe, expect, it, vi } from 'vitest'; +import type { ExecInPod } from '../src/exec.js'; +import type { K8sSandboxConfig } from '../src/config.js'; +import { OUTPUT_TRUNCATED_MARKER } from '../src/transport.js'; import { createPodReadOps, createPodWriteOps, @@ -9,14 +9,14 @@ import { createPodBashOps, createPodLsOps, createPodFindOps, -} from "../src/operations.js"; +} from '../src/operations.js'; const cfg: K8sSandboxConfig = { - pod: "sbx-0", - namespace: "default", + pod: 'sbx-0', + namespace: 'default', context: undefined, - podCwd: "/workspace", - headCwd: "/head", + podCwd: '/workspace', + headCwd: '/head', }; /** Build a fake ExecInPod that returns scripted results and records calls. */ @@ -26,226 +26,233 @@ function fakeExec(result: { stdout?: string; exitCode?: number | null; truncated calls.push({ command, stdin: opts?.stdin?.toString() }); // `?? 0` would swallow a deliberate null (the truncation signal), so branch on undefined. const exitCode = result.exitCode === undefined ? 0 : result.exitCode; - return { stdout: Buffer.from(result.stdout ?? ""), exitCode, truncated: result.truncated ?? false }; + return { + stdout: Buffer.from(result.stdout ?? ''), + exitCode, + truncated: result.truncated ?? false, + }; }; return { fn, calls }; } -describe("read ops", () => { - it("reads a file via cat with the mapped path", async () => { - const { fn, calls } = fakeExec({ stdout: "hello" }); +describe('read ops', () => { + it('reads a file via cat with the mapped path', async () => { + const { fn, calls } = fakeExec({ stdout: 'hello' }); const ops = createPodReadOps(fn, cfg); - const buf = await ops.readFile("/head/a.txt"); - expect(buf.toString()).toBe("hello"); + const buf = await ops.readFile('/head/a.txt'); + expect(buf.toString()).toBe('hello'); expect(calls[0].command).toBe("cat '/workspace/a.txt'"); }); - it("readFile refuses a truncated read instead of returning partial bytes", async () => { + it('readFile refuses a truncated read instead of returning partial bytes', async () => { // The seam signals a cap trip via truncated: true (spec §8). Pi's Edit tool writes // back whatever readFile returns, so returning these bytes would truncate the file // in the sandbox AND write "[output truncated]" into it. const { fn } = fakeExec({ - stdout: "a".repeat(64) + OUTPUT_TRUNCATED_MARKER, + stdout: 'a'.repeat(64) + OUTPUT_TRUNCATED_MARKER, exitCode: null, truncated: true, }); const ops = createPodReadOps(fn, cfg); - await expect(ops.readFile("/head/big.txt")).rejects.toThrow(/output cap.*big\.txt/); + await expect(ops.readFile('/head/big.txt')).rejects.toThrow(/output cap.*big\.txt/); }); - it("readFile rejects when the cat itself fails", async () => { - const { fn } = fakeExec({ stdout: "", exitCode: 1 }); + it('readFile rejects when the cat itself fails', async () => { + const { fn } = fakeExec({ stdout: '', exitCode: 1 }); const ops = createPodReadOps(fn, cfg); - await expect(ops.readFile("/head/missing.txt")).rejects.toThrow(/Read failed in pod.*missing\.txt/); + await expect(ops.readFile('/head/missing.txt')).rejects.toThrow( + /Read failed in pod.*missing\.txt/, + ); }); - it("readFile names the cap and the file size, and suggests a range read", async () => { + it('readFile names the cap and the file size, and suggests a range read', async () => { // pi-fork's read.ts:277 reads the WHOLE file before applying offset/limit, so a // capped file is unreachable through Read even with paging — the model needs to be // told to use bash instead, and how big the file is so it can pick a range. const calls: string[] = []; const fn: ExecInPod = async (command) => { calls.push(command); - if (command.startsWith("stat")) return { stdout: Buffer.from("21000000\n"), exitCode: 0, truncated: false }; - return { stdout: Buffer.from("partial"), exitCode: null, truncated: true }; + if (command.startsWith('stat')) + return { stdout: Buffer.from('21000000\n'), exitCode: 0, truncated: false }; + return { stdout: Buffer.from('partial'), exitCode: null, truncated: true }; }; const ops = createPodReadOps(fn, cfg); // Capture once rather than re-invoking per assertion: each call would re-issue the // stat, and asserting three separate rejections of the same call is not possible. - const err = (await ops.readFile("/head/big.json").catch((e) => e)) as Error; + const err = (await ops.readFile('/head/big.json').catch((e) => e)) as Error; expect(err.message).toMatch(/exceeds the .* output cap/); expect(err.message).toMatch(/21000000/); // the size, so the model can pick a range expect(err.message).toMatch(/sed -n/); // the escape hatch - expect(calls.some((c) => c.startsWith("stat -c %s"))).toBe(true); + expect(calls.some((c) => c.startsWith('stat -c %s'))).toBe(true); }); - it("readFile still reports a signalled cat distinctly from a cap trip", async () => { + it('readFile still reports a signalled cat distinctly from a cap trip', async () => { // truncated: false with a null code is 'no exit status, and NOT our cap'. Blaming // the cap here would send the model chasing a size limit that is not the problem. - const { fn } = fakeExec({ stdout: "", exitCode: null, truncated: false }); + const { fn } = fakeExec({ stdout: '', exitCode: null, truncated: false }); const ops = createPodReadOps(fn, cfg); - const err = (await ops.readFile("/head/a.txt").catch((e) => e)) as Error; + const err = (await ops.readFile('/head/a.txt').catch((e) => e)) as Error; expect(err.message).toMatch(/no exit status/); expect(err.message).not.toMatch(/output cap/); // must not blame a size limit }); - it("readFile survives a stat that fails, omitting the size", async () => { + it('readFile survives a stat that fails, omitting the size', async () => { const fn: ExecInPod = async (command) => - command.startsWith("stat") - ? { stdout: Buffer.from(""), exitCode: 1, truncated: false } - : { stdout: Buffer.from("partial"), exitCode: null, truncated: true }; + command.startsWith('stat') + ? { stdout: Buffer.from(''), exitCode: 1, truncated: false } + : { stdout: Buffer.from('partial'), exitCode: null, truncated: true }; const ops = createPodReadOps(fn, cfg); - await expect(ops.readFile("/head/big.json")).rejects.toThrow(/exceeds the .* output cap/); + await expect(ops.readFile('/head/big.json')).rejects.toThrow(/exceeds the .* output cap/); }); - it("access rejects when test -r exits non-zero", async () => { + it('access rejects when test -r exits non-zero', async () => { const { fn } = fakeExec({ exitCode: 1 }); const ops = createPodReadOps(fn, cfg); - await expect(ops.access("/head/a.txt")).rejects.toThrow(); + await expect(ops.access('/head/a.txt')).rejects.toThrow(); }); - it("detectImageMimeType returns the type for an image, null otherwise", async () => { - const img = createPodReadOps(fakeExec({ stdout: "image/png\n" }).fn, cfg); - expect(await img.detectImageMimeType!("/head/x.png")).toBe("image/png"); - const txt = createPodReadOps(fakeExec({ stdout: "text/plain\n" }).fn, cfg); - expect(await txt.detectImageMimeType!("/head/x.txt")).toBeNull(); + it('detectImageMimeType returns the type for an image, null otherwise', async () => { + const img = createPodReadOps(fakeExec({ stdout: 'image/png\n' }).fn, cfg); + expect(await img.detectImageMimeType!('/head/x.png')).toBe('image/png'); + const txt = createPodReadOps(fakeExec({ stdout: 'text/plain\n' }).fn, cfg); + expect(await txt.detectImageMimeType!('/head/x.txt')).toBeNull(); }); }); -describe("write ops", () => { - it("writes via base64 -d on stdin", async () => { +describe('write ops', () => { + it('writes via base64 -d on stdin', async () => { const { fn, calls } = fakeExec({}); const ops = createPodWriteOps(fn, cfg); - await ops.writeFile("/head/a.txt", "hi"); + await ops.writeFile('/head/a.txt', 'hi'); expect(calls[0].command).toBe("base64 -d > '/workspace/a.txt'"); - expect(calls[0].stdin).toBe(Buffer.from("hi").toString("base64")); + expect(calls[0].stdin).toBe(Buffer.from('hi').toString('base64')); }); - it("mkdir -p with the mapped dir", async () => { + it('mkdir -p with the mapped dir', async () => { const { fn, calls } = fakeExec({}); - await createPodWriteOps(fn, cfg).mkdir("/head/sub"); + await createPodWriteOps(fn, cfg).mkdir('/head/sub'); expect(calls[0].command).toBe("mkdir -p '/workspace/sub'"); }); }); -describe("edit ops", () => { - it("access requires read AND write", async () => { +describe('edit ops', () => { + it('access requires read AND write', async () => { const { fn, calls } = fakeExec({}); - await createPodEditOps(fn, cfg).access("/head/a.txt"); + await createPodEditOps(fn, cfg).access('/head/a.txt'); expect(calls[0].command).toBe("test -r '/workspace/a.txt' && test -w '/workspace/a.txt'"); }); }); -describe("bash ops", () => { - it("cds into the mapped cwd then runs the command, returning exitCode", async () => { +describe('bash ops', () => { + it('cds into the mapped cwd then runs the command, returning exitCode', async () => { const { fn, calls } = fakeExec({ exitCode: 0 }); const ops = createPodBashOps(fn, cfg); const onData = vi.fn(); - const r = await ops.exec("echo hi", "/head", { onData }); + const r = await ops.exec('echo hi', '/head', { onData }); expect(calls[0].command).toBe("cd '/workspace' && echo hi"); expect(r).toEqual({ exitCode: 0 }); }); - it("injects env as a non-leaking, per-invocation prefix when env is provided", async () => { + it('injects env as a non-leaking, per-invocation prefix when env is provided', async () => { const { fn, calls } = fakeExec({ exitCode: 0 }); const ops = createPodBashOps(fn, cfg); - await ops.exec("echo $FOO", "/head", { onData: vi.fn(), env: { FOO: "bar baz" } }); + await ops.exec('echo $FOO', '/head', { onData: vi.fn(), env: { FOO: 'bar baz' } }); expect(calls[0].command).toBe("cd '/workspace' && env FOO='bar baz' bash -c 'echo $FOO'"); }); - it("skips env keys whose value is undefined", async () => { + it('skips env keys whose value is undefined', async () => { const { fn, calls } = fakeExec({ exitCode: 0 }); const ops = createPodBashOps(fn, cfg); - await ops.exec("true", "/head", { onData: vi.fn(), env: { A: "1", B: undefined } }); + await ops.exec('true', '/head', { onData: vi.fn(), env: { A: '1', B: undefined } }); expect(calls[0].command).toBe("cd '/workspace' && env A='1' bash -c 'true'"); }); - it("emits the M2 form (no prefix) when env is absent or empty", async () => { + it('emits the M2 form (no prefix) when env is absent or empty', async () => { const { fn, calls } = fakeExec({ exitCode: 0 }); const ops = createPodBashOps(fn, cfg); - await ops.exec("echo hi", "/head", { onData: vi.fn(), env: {} }); + await ops.exec('echo hi', '/head', { onData: vi.fn(), env: {} }); expect(calls[0].command).toBe("cd '/workspace' && echo hi"); }); - it("drops env keys that are not valid POSIX names (no injection)", async () => { + it('drops env keys that are not valid POSIX names (no injection)', async () => { const { fn, calls } = fakeExec({ exitCode: 0 }); const ops = createPodBashOps(fn, cfg); - await ops.exec("true", "/head", { + await ops.exec('true', '/head', { onData: vi.fn(), - env: { GOOD: "1", "BAD KEY": "x", "PATH=/evil; rm -rf /": "y" }, + env: { GOOD: '1', 'BAD KEY': 'x', 'PATH=/evil; rm -rf /': 'y' }, }); expect(calls[0].command).toBe("cd '/workspace' && env GOOD='1' bash -c 'true'"); }); - it("reports a cap-truncated command as exit 137 rather than success", async () => { + it('reports a cap-truncated command as exit 137 rather than success', async () => { // Pi's bash tool treats a null exit code as non-failing // (pi-fork/.../tools/bash.ts:397: `exitCode !== 0 && exitCode !== null`), so // passing the seam's null through told the model a SIGKILLed flood had completed // normally (#181). 137 is 128+9, the conventional SIGKILL status — not a // fabricated code: the command really was killed by signal 9 at the cap. Pi then // throws with the streamed output tail attached, so the model gets both facts. - const { fn } = fakeExec({ stdout: "", exitCode: null, truncated: true }); + const { fn } = fakeExec({ stdout: '', exitCode: null, truncated: true }); const ops = createPodBashOps(fn, cfg); - const r = await ops.exec("yes", "/head", { onData: () => {} }); + const r = await ops.exec('yes', '/head', { onData: () => {} }); expect(r.exitCode).toBe(137); }); - it("passes a null exit code through untouched when it is NOT a cap trip", async () => { + it('passes a null exit code through untouched when it is NOT a cap trip', async () => { // truncated: false with a null code means "signalled, no status" — not our cap. // Mapping that to 137 too would invent a cause, so it stays null and Pi keeps // treating it as it does today. - const { fn } = fakeExec({ stdout: "", exitCode: null, truncated: false }); + const { fn } = fakeExec({ stdout: '', exitCode: null, truncated: false }); const ops = createPodBashOps(fn, cfg); - const r = await ops.exec("something-signalled", "/head", { onData: () => {} }); + const r = await ops.exec('something-signalled', '/head', { onData: () => {} }); expect(r.exitCode).toBeNull(); }); - it("passes a genuine non-zero exit code through unchanged", async () => { - const { fn } = fakeExec({ stdout: "", exitCode: 3, truncated: false }); + it('passes a genuine non-zero exit code through unchanged', async () => { + const { fn } = fakeExec({ stdout: '', exitCode: 3, truncated: false }); const ops = createPodBashOps(fn, cfg); - const r = await ops.exec("false", "/head", { onData: () => {} }); + const r = await ops.exec('false', '/head', { onData: () => {} }); expect(r.exitCode).toBe(3); }); }); -describe("ls ops", () => { - it("readdir splits lines and drops blanks", async () => { - const { fn, calls } = fakeExec({ stdout: "a.txt\nb\n\n" }); - const entries = await createPodLsOps(fn, cfg).readdir("/head"); - expect(entries).toEqual(["a.txt", "b"]); +describe('ls ops', () => { + it('readdir splits lines and drops blanks', async () => { + const { fn, calls } = fakeExec({ stdout: 'a.txt\nb\n\n' }); + const entries = await createPodLsOps(fn, cfg).readdir('/head'); + expect(entries).toEqual(['a.txt', 'b']); expect(calls[0].command).toBe("ls -1A '/workspace'"); }); - it("stat reports directory vs file", async () => { - const dir = await createPodLsOps(fakeExec({ stdout: "DIR\n" }).fn, cfg).stat("/head/d"); + it('stat reports directory vs file', async () => { + const dir = await createPodLsOps(fakeExec({ stdout: 'DIR\n' }).fn, cfg).stat('/head/d'); expect((await dir).isDirectory()).toBe(true); - const file = await createPodLsOps(fakeExec({ stdout: "FILE\n" }).fn, cfg).stat("/head/f"); + const file = await createPodLsOps(fakeExec({ stdout: 'FILE\n' }).fn, cfg).stat('/head/f'); expect((await file).isDirectory()).toBe(false); }); - it("readdir refuses a truncated listing instead of returning a partial list with the marker as an entry", async () => { + it('readdir refuses a truncated listing instead of returning a partial list with the marker as an entry', async () => { // A cap trip (e.g. a directory with ~200k entries) means what came back is not a // trustworthy directory listing — it may even contain OUTPUT_TRUNCATED_MARKER as a // bogus "entry". const { fn } = fakeExec({ - stdout: "a.txt\nb.txt\n" + OUTPUT_TRUNCATED_MARKER, + stdout: 'a.txt\nb.txt\n' + OUTPUT_TRUNCATED_MARKER, exitCode: null, truncated: true, }); const ops = createPodLsOps(fn, cfg); - await expect(ops.readdir("/head/big-dir")).rejects.toThrow(/output cap.*big-dir/); + await expect(ops.readdir('/head/big-dir')).rejects.toThrow(/output cap.*big-dir/); }); }); -describe("find ops", () => { - it("globs via rg --files, honouring the ignore list and stripping ./", async () => { - const { fn, calls } = fakeExec({ stdout: "src/a.ts\nb.ts\n" }); +describe('find ops', () => { + it('globs via rg --files, honouring the ignore list and stripping ./', async () => { + const { fn, calls } = fakeExec({ stdout: 'src/a.ts\nb.ts\n' }); const ops = createPodFindOps(fn, cfg); - const results = await ops.glob("*.ts", "/head", { - ignore: ["**/node_modules/**", "**/.git/**"], + const results = await ops.glob('*.ts', '/head', { + ignore: ['**/node_modules/**', '**/.git/**'], limit: 100, }); - expect(results).toEqual(["src/a.ts", "b.ts"]); + expect(results).toEqual(['src/a.ts', 'b.ts']); expect(calls[0].command).toBe( "cd '/workspace' && rg --files --hidden -g '*.ts' " + "-g '!**/node_modules/**' -g '!**/.git/**' | head -n 100; " + @@ -253,61 +260,61 @@ describe("find ops", () => { ); }); - it("emits no ignore globs when the ignore list is empty", async () => { - const { fn, calls } = fakeExec({ stdout: "" }); + it('emits no ignore globs when the ignore list is empty', async () => { + const { fn, calls } = fakeExec({ stdout: '' }); const ops = createPodFindOps(fn, cfg); - await ops.glob("*.go", "/head", { ignore: [], limit: 50 }); + await ops.glob('*.go', '/head', { ignore: [], limit: 50 }); expect(calls[0].command).toBe( "cd '/workspace' && rg --files --hidden -g '*.go' | head -n 50; " + 'rc=${PIPESTATUS[0]}; [ "$rc" = 0 ] || [ "$rc" = 1 ] || [ "$rc" = 141 ] || exit "$rc"', ); }); - it("returns an empty result when ripgrep finds no matches", async () => { - const { fn } = fakeExec({ stdout: "", exitCode: 1 }); + it('returns an empty result when ripgrep finds no matches', async () => { + const { fn } = fakeExec({ stdout: '', exitCode: 1 }); const ops = createPodFindOps(fn, cfg); - await expect(ops.glob("*.go", "/head", { ignore: [], limit: 50 })).resolves.toEqual([]); + await expect(ops.glob('*.go', '/head', { ignore: [], limit: 50 })).resolves.toEqual([]); }); - it("rejects a ripgrep failure instead of returning an empty result", async () => { + it('rejects a ripgrep failure instead of returning an empty result', async () => { const { fn } = fakeExec({ exitCode: 2 }); const ops = createPodFindOps(fn, cfg); - await expect(ops.glob("[", "/head", { ignore: [], limit: 50 })).rejects.toThrow( - "glob failed in pod", + await expect(ops.glob('[', '/head', { ignore: [], limit: 50 })).rejects.toThrow( + 'glob failed in pod', ); }); - it("glob refuses a truncated listing instead of returning a partial list with the marker as a path", async () => { + it('glob refuses a truncated listing instead of returning a partial list with the marker as a path', async () => { // A cap trip means what came back is not a trustworthy file list — it may even // contain OUTPUT_TRUNCATED_MARKER as a bogus "path" entry. const { fn } = fakeExec({ - stdout: "src/a.ts\nb.ts\n" + OUTPUT_TRUNCATED_MARKER, + stdout: 'src/a.ts\nb.ts\n' + OUTPUT_TRUNCATED_MARKER, exitCode: null, truncated: true, }); const ops = createPodFindOps(fn, cfg); - await expect(ops.glob("*.ts", "/head", { ignore: [], limit: 5 })).rejects.toThrow( - /output cap/, - ); + await expect(ops.glob('*.ts', '/head', { ignore: [], limit: 5 })).rejects.toThrow(/output cap/); }); - it("glob rejects when rg itself fails (e.g. a bad pattern) instead of returning an empty list", async () => { - const { fn } = fakeExec({ stdout: "", exitCode: 2 }); + it('glob rejects when rg itself fails (e.g. a bad pattern) instead of returning an empty list', async () => { + const { fn } = fakeExec({ stdout: '', exitCode: 2 }); const ops = createPodFindOps(fn, cfg); - await expect(ops.glob("[", "/head", { ignore: [], limit: 100 })).rejects.toThrow(/glob failed in pod/); + await expect(ops.glob('[', '/head', { ignore: [], limit: 100 })).rejects.toThrow( + /glob failed in pod/, + ); }); - it("glob reports a cap trip distinctly from an rg failure", async () => { - const { fn } = fakeExec({ stdout: "a.ts\n", exitCode: null, truncated: true }); + it('glob reports a cap trip distinctly from an rg failure', async () => { + const { fn } = fakeExec({ stdout: 'a.ts\n', exitCode: null, truncated: true }); const ops = createPodFindOps(fn, cfg); - await expect(ops.glob("*.ts", "/head", { ignore: [], limit: 100 })).rejects.toThrow( + await expect(ops.glob('*.ts', '/head', { ignore: [], limit: 100 })).rejects.toThrow( /output cap/, ); }); - it("exists uses test -e on the mapped path", async () => { + it('exists uses test -e on the mapped path', async () => { const { fn, calls } = fakeExec({ exitCode: 0 }); - expect(await createPodFindOps(fn, cfg).exists("/head/x")).toBe(true); + expect(await createPodFindOps(fn, cfg).exists('/head/x')).toBe(true); expect(calls[0].command).toBe("test -e '/workspace/x'"); }); }); diff --git a/packages/k8s-sandbox/test/output-cap-coupling.test.ts b/packages/k8s-sandbox/test/output-cap-coupling.test.ts index d09255e..138f867 100644 --- a/packages/k8s-sandbox/test/output-cap-coupling.test.ts +++ b/packages/k8s-sandbox/test/output-cap-coupling.test.ts @@ -1,11 +1,11 @@ -import { readFileSync } from "node:fs"; -import { fileURLToPath } from "node:url"; -import { dirname, resolve } from "node:path"; -import { describe, expect, it } from "vitest"; -import { DEFAULT_OUTPUT_CAP } from "../src/transport.js"; +import { readFileSync } from 'node:fs'; +import { fileURLToPath } from 'node:url'; +import { dirname, resolve } from 'node:path'; +import { describe, expect, it } from 'vitest'; +import { DEFAULT_OUTPUT_CAP } from '../src/transport.js'; -const REPO_ROOT = resolve(dirname(fileURLToPath(import.meta.url)), "../../.."); -const RUNNER_GO = resolve(REPO_ROOT, "remote-worker/internal/exec/runner.go"); +const REPO_ROOT = resolve(dirname(fileURLToPath(import.meta.url)), '../../..'); +const RUNNER_GO = resolve(REPO_ROOT, 'remote-worker/internal/exec/runner.go'); /** * Loud-throw reader (same shape as knative-server's worker-deployment test): a @@ -13,17 +13,17 @@ const RUNNER_GO = resolve(REPO_ROOT, "remote-worker/internal/exec/runner.go"); * as NaN and silently pass the comparison. */ const readBufferCapBytes = (): number => { - const runnerGo = readFileSync(RUNNER_GO, "utf8"); + const runnerGo = readFileSync(RUNNER_GO, 'utf8'); const match = /BufferCap = (\d+) \* 1024 \* 1024/.exec(runnerGo); if (!match) { throw new Error( - "could not find `BufferCap = N * 1024 * 1024` in runner.go — constant renamed or reformatted?", + 'could not find `BufferCap = N * 1024 * 1024` in runner.go — constant renamed or reformatted?', ); } return Number(match[1]) * 1024 * 1024; }; -describe("output cap is pinned across the language boundary", () => { +describe('output cap is pinned across the language boundary', () => { it("DEFAULT_OUTPUT_CAP equals the Go worker's BufferCap", () => { // transport.ts says the Go worker's BufferCap "is pinned to this value — change // one and change the other". Nothing enforced that, so the two could drift: a diff --git a/packages/k8s-sandbox/test/paths.test.ts b/packages/k8s-sandbox/test/paths.test.ts index fa3e492..fc4eddf 100644 --- a/packages/k8s-sandbox/test/paths.test.ts +++ b/packages/k8s-sandbox/test/paths.test.ts @@ -1,23 +1,25 @@ -import { describe, expect, it } from "vitest"; -import { mapPath, shQuote } from "../src/paths.js"; +import { describe, expect, it } from 'vitest'; +import { mapPath, shQuote } from '../src/paths.js'; -describe("shQuote", () => { - it("wraps a plain path in single quotes", () => { - expect(shQuote("/workspace/a.txt")).toBe("'/workspace/a.txt'"); +describe('shQuote', () => { + it('wraps a plain path in single quotes', () => { + expect(shQuote('/workspace/a.txt')).toBe("'/workspace/a.txt'"); }); - it("escapes embedded single quotes", () => { + it('escapes embedded single quotes', () => { expect(shQuote("it's")).toBe("'it'\\''s'"); }); }); -describe("mapPath", () => { - it("rewrites a head-cwd prefix to the pod cwd", () => { - expect(mapPath("/Users/dev/proj/src/a.ts", "/Users/dev/proj", "/workspace")).toBe("/workspace/src/a.ts"); +describe('mapPath', () => { + it('rewrites a head-cwd prefix to the pod cwd', () => { + expect(mapPath('/Users/dev/proj/src/a.ts', '/Users/dev/proj', '/workspace')).toBe( + '/workspace/src/a.ts', + ); }); - it("rewrites the head cwd itself", () => { - expect(mapPath("/Users/dev/proj", "/Users/dev/proj", "/workspace")).toBe("/workspace"); + it('rewrites the head cwd itself', () => { + expect(mapPath('/Users/dev/proj', '/Users/dev/proj', '/workspace')).toBe('/workspace'); }); - it("leaves paths outside the head cwd untouched", () => { - expect(mapPath("/etc/hosts", "/Users/dev/proj", "/workspace")).toBe("/etc/hosts"); + it('leaves paths outside the head cwd untouched', () => { + expect(mapPath('/etc/hosts', '/Users/dev/proj', '/workspace')).toBe('/etc/hosts'); }); }); diff --git a/packages/k8s-sandbox/test/persistent-exec-conformance.test.ts b/packages/k8s-sandbox/test/persistent-exec-conformance.test.ts index 12d8c06..ce7eb7c 100644 --- a/packages/k8s-sandbox/test/persistent-exec-conformance.test.ts +++ b/packages/k8s-sandbox/test/persistent-exec-conformance.test.ts @@ -1,18 +1,18 @@ -import { EventEmitter } from "node:events"; -import { vi } from "vitest"; -import type { K8sSandboxConfig } from "../src/config.js"; -import type { ExecInPod } from "../src/transport.js"; -import { persistentExecInPod } from "../src/persistent-exec.js"; -import { runConformance, type TransportFactory } from "./conformance.js"; +import { EventEmitter } from 'node:events'; +import { vi } from 'vitest'; +import type { K8sSandboxConfig } from '../src/config.js'; +import type { ExecInPod } from '../src/transport.js'; +import { persistentExecInPod } from '../src/persistent-exec.js'; +import { runConformance, type TransportFactory } from './conformance.js'; -const SOH = "\x01"; +const SOH = '\x01'; const cfg: K8sSandboxConfig = { - pod: "sbx-0", - namespace: "team1", + pod: 'sbx-0', + namespace: 'team1', context: undefined, - podCwd: "/workspace", - headCwd: "/head", + podCwd: '/workspace', + headCwd: '/head', }; /** @@ -31,26 +31,26 @@ const persistentFactory: TransportFactory = (b, opts) => { const child = new EventEmitter() as any; child.stdout = new EventEmitter(); child.stderr = new EventEmitter(); - child.kill = vi.fn(() => child.emit("close", null)); + child.kill = vi.fn(() => child.emit('close', null)); child.stdin = { write: (line: string) => { // The pod-side cap is the mechanism here, so its presence in the framed command is // the witness the battery asserts against the declared `producer-side-cap`. if (line.includes(`| head -c ${cap + 1} |`)) capStageSeen = true; // wrapCommand emits: printf '\x01B%s\n' ; { … } | head -c N | base64; … - const n = line.match(/printf '\x01B%s\\n' (\S+);/)?.[1] ?? "n1"; + const n = line.match(/printf '\x01B%s\\n' (\S+);/)?.[1] ?? 'n1'; // stdin rides in a nonce-delimited heredoc, emitted as latin1. const hd = line.match(/<<'KAGENTI_EOF_[^']+'\n([\s\S]*?)\nKAGENTI_EOF_/); - if (hd) stdinSeen = Buffer.from(hd[1], "latin1"); + if (hd) stdinSeen = Buffer.from(hd[1], 'latin1'); if (b.hang) return true; // Emit stderr chunks nowhere: this transport does not stream (streams: false). - const raw = Buffer.from((b.stdout ?? []).join("")); + const raw = Buffer.from((b.stdout ?? []).join('')); const capped = raw.subarray(0, cap + 1); // what `head -c ` would yield const code = capped.length < raw.length ? 141 : (b.exitCode ?? 0); // 141 = SIGPIPE queueMicrotask(() => { child.stdout.emit( - "data", - Buffer.from(`${SOH}B${n}\n${capped.toString("base64")}\n${SOH}E${n} ${code}\n`), + 'data', + Buffer.from(`${SOH}B${n}\n${capped.toString('base64')}\n${SOH}E${n} ${code}\n`), ); }); return true; @@ -61,7 +61,7 @@ const persistentFactory: TransportFactory = (b, opts) => { // A fallback that throws: reaching it means the transport treated a cap trip as a dead // channel and re-ran the command, which is the specific regression Task 3 guards. const fallback: ExecInPod = async () => { - throw new Error("fallback must not be reached during conformance"); + throw new Error('fallback must not be reached during conformance'); }; const transport = persistentExecInPod(cfg, { fallback, @@ -71,11 +71,11 @@ const persistentFactory: TransportFactory = (b, opts) => { return { transport, stdinSeen: () => stdinSeen, - producerStop: () => (capStageSeen ? "producer-side-cap" : "none"), + producerStop: () => (capStageSeen ? 'producer-side-cap' : 'none'), }; }; -runConformance("persistentExecInPod", persistentFactory, { - producerStop: "producer-side-cap", +runConformance('persistentExecInPod', persistentFactory, { + producerStop: 'producer-side-cap', streams: false, }); diff --git a/packages/k8s-sandbox/test/persistent-exec.test.ts b/packages/k8s-sandbox/test/persistent-exec.test.ts index 8d3d041..a9b21cc 100644 --- a/packages/k8s-sandbox/test/persistent-exec.test.ts +++ b/packages/k8s-sandbox/test/persistent-exec.test.ts @@ -1,22 +1,22 @@ -import { EventEmitter } from "node:events"; -import { afterEach, describe, expect, it, vi } from "vitest"; -import type { K8sSandboxConfig } from "../src/config.js"; -import type { ExecInPod } from "../src/transport.js"; -import { OUTPUT_TRUNCATED_MARKER } from "../src/transport.js"; -import { CAP_STAGE_FAILED } from "../src/framing.js"; -import { buildPersistentKubectlArgs, persistentExecInPod } from "../src/persistent-exec.js"; +import { EventEmitter } from 'node:events'; +import { afterEach, describe, expect, it, vi } from 'vitest'; +import type { K8sSandboxConfig } from '../src/config.js'; +import type { ExecInPod } from '../src/transport.js'; +import { OUTPUT_TRUNCATED_MARKER } from '../src/transport.js'; +import { CAP_STAGE_FAILED } from '../src/framing.js'; +import { buildPersistentKubectlArgs, persistentExecInPod } from '../src/persistent-exec.js'; const cfg: K8sSandboxConfig = { - pod: "sbx-0", - namespace: "team1", + pod: 'sbx-0', + namespace: 'team1', context: undefined, - podCwd: "/workspace", - headCwd: "/head", + podCwd: '/workspace', + headCwd: '/head', }; -const SOH = "\x01"; +const SOH = '\x01'; function frameFor(nonce: string, payload: string, code = 0): Buffer { - const b64 = Buffer.from(payload).toString("base64"); + const b64 = Buffer.from(payload).toString('base64'); return Buffer.from(`${SOH}B${nonce}\n${b64}\n${SOH}E${nonce} ${code}\n`); } @@ -27,7 +27,7 @@ function makeFakeChild() { child.stderr = new EventEmitter(); const writes: string[] = []; child.stdin = { write: (s: string) => (writes.push(s), true), end: vi.fn() }; - child.kill = vi.fn(() => child.emit("close", null)); + child.kill = vi.fn(() => child.emit('close', null)); return { child, writes }; } @@ -41,7 +41,7 @@ function fakeSpawn(children: any[]) { return { spawn, calls }; } -function recordingFallback(result = { stdout: Buffer.from("FB"), exitCode: 7, truncated: false }) { +function recordingFallback(result = { stdout: Buffer.from('FB'), exitCode: 7, truncated: false }) { const calls: string[] = []; const fallback: ExecInPod = async (command) => (calls.push(command), result); return { fallback, calls }; @@ -49,21 +49,35 @@ function recordingFallback(result = { stdout: Buffer.from("FB"), exitCode: 7, tr afterEach(() => vi.useRealTimers()); -describe("buildPersistentKubectlArgs", () => { - it("execs a bare interactive bash (no -c) with namespace", () => { +describe('buildPersistentKubectlArgs', () => { + it('execs a bare interactive bash (no -c) with namespace', () => { expect(buildPersistentKubectlArgs(cfg)).toEqual([ - "exec", "-i", "-n", "team1", "sbx-0", "--", "bash", + 'exec', + '-i', + '-n', + 'team1', + 'sbx-0', + '--', + 'bash', ]); }); - it("includes --context when set", () => { - expect(buildPersistentKubectlArgs({ ...cfg, context: "kind-x" })).toEqual([ - "exec", "-i", "-n", "team1", "--context", "kind-x", "sbx-0", "--", "bash", + it('includes --context when set', () => { + expect(buildPersistentKubectlArgs({ ...cfg, context: 'kind-x' })).toEqual([ + 'exec', + '-i', + '-n', + 'team1', + '--context', + 'kind-x', + 'sbx-0', + '--', + 'bash', ]); }); }); -describe("persistentExecInPod", () => { - it("spawns once, writes the framed command, resolves from the matching frame", async () => { +describe('persistentExecInPod', () => { + it('spawns once, writes the framed command, resolves from the matching frame', async () => { const { child, writes } = makeFakeChild(); const { spawn, calls } = fakeSpawn([child]); const { fallback } = recordingFallback(); @@ -72,59 +86,59 @@ describe("persistentExecInPod", () => { const p = t.exec("cat '/workspace/a.txt'"); expect(calls).toHaveLength(1); // lazy spawn happened expect(writes[0]).toContain("cat '/workspace/a.txt'"); - child.stdout.emit("data", frameFor("n1", "hello", 0)); - expect(await p).toEqual({ stdout: Buffer.from("hello"), exitCode: 0, truncated: false }); + child.stdout.emit('data', frameFor('n1', 'hello', 0)); + expect(await p).toEqual({ stdout: Buffer.from('hello'), exitCode: 0, truncated: false }); }); - it("reuses one child across sequential calls (no second spawn)", async () => { + it('reuses one child across sequential calls (no second spawn)', async () => { const { child } = makeFakeChild(); const { spawn, calls } = fakeSpawn([child]); const t = persistentExecInPod(cfg, { fallback: recordingFallback().fallback, spawn }); - const p1 = t.exec("echo a"); - child.stdout.emit("data", frameFor("n1", "a", 0)); + const p1 = t.exec('echo a'); + child.stdout.emit('data', frameFor('n1', 'a', 0)); await p1; - const p2 = t.exec("echo b"); - child.stdout.emit("data", frameFor("n2", "b", 0)); + const p2 = t.exec('echo b'); + child.stdout.emit('data', frameFor('n2', 'b', 0)); await p2; expect(calls).toHaveLength(1); }); - it("serializes: the second command is not written until the first frame arrives", async () => { + it('serializes: the second command is not written until the first frame arrives', async () => { const { child, writes } = makeFakeChild(); const { spawn } = fakeSpawn([child]); const t = persistentExecInPod(cfg, { fallback: recordingFallback().fallback, spawn }); - const p1 = t.exec("first"); - const p2 = t.exec("second"); + const p1 = t.exec('first'); + const p2 = t.exec('second'); expect(writes).toHaveLength(1); // only first in flight - child.stdout.emit("data", frameFor("n1", "", 0)); + child.stdout.emit('data', frameFor('n1', '', 0)); await p1; expect(writes).toHaveLength(2); // second now written - child.stdout.emit("data", frameFor("n2", "", 0)); + child.stdout.emit('data', frameFor('n2', '', 0)); await p2; }); - it("falls back when the session dies mid-command", async () => { + it('falls back when the session dies mid-command', async () => { const { child } = makeFakeChild(); const { spawn } = fakeSpawn([child, makeFakeChild().child]); const { fallback, calls } = recordingFallback(); const t = persistentExecInPod(cfg, { fallback, spawn }); const p = t.exec("cat '/workspace/a.txt'"); - child.emit("error", new Error("broken pipe")); - expect(await p).toEqual({ stdout: Buffer.from("FB"), exitCode: 7, truncated: false }); + child.emit('error', new Error('broken pipe')); + expect(await p).toEqual({ stdout: Buffer.from('FB'), exitCode: 7, truncated: false }); expect(calls).toEqual(["cat '/workspace/a.txt'"]); }); - it("times out: kills the child and rejects with timeout:", async () => { + it('times out: kills the child and rejects with timeout:', async () => { vi.useFakeTimers(); const { child } = makeFakeChild(); const { spawn } = fakeSpawn([child]); const t = persistentExecInPod(cfg, { fallback: recordingFallback().fallback, spawn }); - const p = t.exec("sleep 999", { timeout: 2 }); - const assertion = expect(p).rejects.toThrow("timeout:2"); + const p = t.exec('sleep 999', { timeout: 2 }); + const assertion = expect(p).rejects.toThrow('timeout:2'); await vi.advanceTimersByTimeAsync(2000); await assertion; expect(child.kill).toHaveBeenCalled(); @@ -136,13 +150,13 @@ describe("persistentExecInPod", () => { const t = persistentExecInPod(cfg, { fallback: recordingFallback().fallback, spawn }); const ac = new AbortController(); - const p = t.exec("sleep 999", { signal: ac.signal }); + const p = t.exec('sleep 999', { signal: ac.signal }); ac.abort(); - await expect(p).rejects.toThrow("aborted"); + await expect(p).rejects.toThrow('aborted'); expect(child.kill).toHaveBeenCalled(); }); - it("a broken cap stage routes to the fallback and latches, without respawning", async () => { + it('a broken cap stage routes to the fallback and latches, without respawning', async () => { // CAP_STAGE_FAILED means our own wrapper pipeline failed, not the command. Unhandled it // arrives as empty stdout with exit 0, which readFile would return as a successful // empty read and Pi's Edit would write back — truncating the file. The channel must @@ -155,12 +169,12 @@ describe("persistentExecInPod", () => { const t = persistentExecInPod(cfg, { fallback, spawn }); const p1 = t.exec("cat '/workspace/a.txt'"); - child.stdout.emit("data", frameFor("n1", "", CAP_STAGE_FAILED)); - expect(await p1).toEqual({ stdout: Buffer.from("FB"), exitCode: 7, truncated: false }); + child.stdout.emit('data', frameFor('n1', '', CAP_STAGE_FAILED)); + expect(await p1).toEqual({ stdout: Buffer.from('FB'), exitCode: 7, truncated: false }); // Latched: the second call goes straight to the fallback, and no second child is spawned. expect(await t.exec("cat '/workspace/b.txt'")).toEqual({ - stdout: Buffer.from("FB"), + stdout: Buffer.from('FB'), exitCode: 7, truncated: false, }); @@ -169,23 +183,27 @@ describe("persistentExecInPod", () => { await t.close(); }); - it("close() kills the child and routes later calls to the fallback", async () => { + it('close() kills the child and routes later calls to the fallback', async () => { const { child } = makeFakeChild(); const { spawn } = fakeSpawn([child]); const { fallback, calls } = recordingFallback(); const t = persistentExecInPod(cfg, { fallback, spawn }); - const p1 = t.exec("echo a"); - child.stdout.emit("data", frameFor("n1", "a", 0)); + const p1 = t.exec('echo a'); + child.stdout.emit('data', frameFor('n1', 'a', 0)); await p1; await t.close(); expect(child.stdin.end).toHaveBeenCalled(); expect(child.kill).toHaveBeenCalled(); - expect(await t.exec("echo later")).toEqual({ stdout: Buffer.from("FB"), exitCode: 7, truncated: false }); - expect(calls).toEqual(["echo later"]); + expect(await t.exec('echo later')).toEqual({ + stdout: Buffer.from('FB'), + exitCode: 7, + truncated: false, + }); + expect(calls).toEqual(['echo later']); }); - it("flags a cap trip, trims to the cap, appends the marker, and does NOT fall back", async () => { + it('flags a cap trip, trims to the cap, appends the marker, and does NOT fall back', async () => { // The critical property. persistentExecInPod's `fail` path retries through // deps.fallback, which extension.ts:52 sets to the now-capped KubectlTransport. If a // cap trip were routed through `fail`, the command would RE-RUN and flood twice @@ -196,9 +214,9 @@ describe("persistentExecInPod", () => { const { fallback, calls } = recordingFallback(); const t = persistentExecInPod(cfg, { fallback, spawn, outputCapBytes: 6 }); - const p = t.exec("cat big"); + const p = t.exec('cat big'); // The pod's `head -c 7` yields cap+1 = 7 bytes; exit 141 because head closed the pipe. - child.stdout.emit("data", frameFor("n1", "aaaabbb", 141)); + child.stdout.emit('data', frameFor('n1', 'aaaabbb', 141)); const r = await p; expect(r.truncated).toBe(true); @@ -207,30 +225,30 @@ describe("persistentExecInPod", () => { expect(calls).toEqual([]); // no fallback, so no re-run }); - it("does not flag output that lands exactly on the cap", async () => { + it('does not flag output that lands exactly on the cap', async () => { const { child } = makeFakeChild(); const { spawn } = fakeSpawn([child]); const { fallback } = recordingFallback(); const t = persistentExecInPod(cfg, { fallback, spawn, outputCapBytes: 6 }); - const p = t.exec("cat exact"); - child.stdout.emit("data", frameFor("n1", "aaaabb", 0)); + const p = t.exec('cat exact'); + child.stdout.emit('data', frameFor('n1', 'aaaabb', 0)); const r = await p; expect(r.truncated).toBe(false); expect(r.exitCode).toBe(0); - expect(r.stdout.toString()).toBe("aaaabb"); + expect(r.stdout.toString()).toBe('aaaabb'); }); - it("writes the cap into the framed command so the pod enforces it", async () => { + it('writes the cap into the framed command so the pod enforces it', async () => { const { child, writes } = makeFakeChild(); const { spawn } = fakeSpawn([child]); const { fallback } = recordingFallback(); const t = persistentExecInPod(cfg, { fallback, spawn, outputCapBytes: 6 }); - const p = t.exec("cat f"); - expect(writes[0]).toContain("| head -c 7 |"); - child.stdout.emit("data", frameFor("n1", "", 0)); + const p = t.exec('cat f'); + expect(writes[0]).toContain('| head -c 7 |'); + child.stdout.emit('data', frameFor('n1', '', 0)); await p; }); }); diff --git a/packages/k8s-sandbox/test/pool.test.ts b/packages/k8s-sandbox/test/pool.test.ts index 2111eb6..b84b9e5 100644 --- a/packages/k8s-sandbox/test/pool.test.ts +++ b/packages/k8s-sandbox/test/pool.test.ts @@ -1,36 +1,47 @@ -import { describe, it, expect } from "vitest"; -import { buildPoolPodsArgs, parsePodNames, listPoolPods } from "../src/pool.js"; +import { describe, it, expect } from 'vitest'; +import { buildPoolPodsArgs, parsePodNames, listPoolPods } from '../src/pool.js'; -describe("buildPoolPodsArgs", () => { - it("lists Running pods by selector with a per-name jsonpath", () => { - expect(buildPoolPodsArgs("app=sandbox", "default")).toEqual([ - "get", "pod", "-n", "default", "-l", "app=sandbox", - "--field-selector=status.phase=Running", - "-o", "jsonpath={range .items[*]}{.metadata.name}{'\\n'}{end}", +describe('buildPoolPodsArgs', () => { + it('lists Running pods by selector with a per-name jsonpath', () => { + expect(buildPoolPodsArgs('app=sandbox', 'default')).toEqual([ + 'get', + 'pod', + '-n', + 'default', + '-l', + 'app=sandbox', + '--field-selector=status.phase=Running', + '-o', + "jsonpath={range .items[*]}{.metadata.name}{'\\n'}{end}", ]); }); - it("adds --context when provided", () => { - expect(buildPoolPodsArgs("app=sandbox", "team1", "kind-x")).toContain("--context"); + it('adds --context when provided', () => { + expect(buildPoolPodsArgs('app=sandbox', 'team1', 'kind-x')).toContain('--context'); }); }); -describe("parsePodNames", () => { - it("splits, trims, and drops blanks", () => { - expect(parsePodNames("sandbox-0-0\nsandbox-1-0\n\n sandbox-2-0 \n")).toEqual([ - "sandbox-0-0", "sandbox-1-0", "sandbox-2-0", +describe('parsePodNames', () => { + it('splits, trims, and drops blanks', () => { + expect(parsePodNames('sandbox-0-0\nsandbox-1-0\n\n sandbox-2-0 \n')).toEqual([ + 'sandbox-0-0', + 'sandbox-1-0', + 'sandbox-2-0', ]); }); - it("returns [] for empty output", () => { - expect(parsePodNames("")).toEqual([]); + it('returns [] for empty output', () => { + expect(parsePodNames('')).toEqual([]); }); }); -describe("listPoolPods", () => { - it("runs the built args and parses the result", async () => { +describe('listPoolPods', () => { + it('runs the built args and parses the result', async () => { const calls: string[][] = []; - const run = async (args: string[]) => { calls.push(args); return "sandbox-0-0\nsandbox-1-0\n"; }; - const pods = await listPoolPods("app=sandbox", "default", undefined, run); - expect(pods).toEqual(["sandbox-0-0", "sandbox-1-0"]); - expect(calls[0]).toEqual(buildPoolPodsArgs("app=sandbox", "default")); + const run = async (args: string[]) => { + calls.push(args); + return 'sandbox-0-0\nsandbox-1-0\n'; + }; + const pods = await listPoolPods('app=sandbox', 'default', undefined, run); + expect(pods).toEqual(['sandbox-0-0', 'sandbox-1-0']); + expect(calls[0]).toEqual(buildPoolPodsArgs('app=sandbox', 'default')); }); }); diff --git a/packages/k8s-sandbox/test/proto-contract.test.ts b/packages/k8s-sandbox/test/proto-contract.test.ts index 39cddb4..26a9748 100644 --- a/packages/k8s-sandbox/test/proto-contract.test.ts +++ b/packages/k8s-sandbox/test/proto-contract.test.ts @@ -1,31 +1,31 @@ -import { describe, it, expect } from "vitest"; -import { Hello, Exec, End } from "../src/gen/sandbox/v1/sandbox"; -import { Chunk, Stream } from "../src/gen/sandbox/v1/sandbox.js"; +import { describe, it, expect } from 'vitest'; +import { Hello, Exec, End } from '../src/gen/sandbox/v1/sandbox'; +import { Chunk, Stream } from '../src/gen/sandbox/v1/sandbox.js'; -describe("sandbox/v1 generated TypeScript stubs", () => { - it("round-trips a Hello through binary encode/decode", () => { +describe('sandbox/v1 generated TypeScript stubs', () => { + it('round-trips a Hello through binary encode/decode', () => { const bytes = Hello.encode({ - sandboxId: "sbx-1", - labels: { team: "alpha" }, - capabilities: ["python3", "kubectl"], - image: "img@sha256:abc", - arch: "amd64", + sandboxId: 'sbx-1', + labels: { team: 'alpha' }, + capabilities: ['python3', 'kubectl'], + image: 'img@sha256:abc', + arch: 'amd64', capacityMax: 4, - trust: "trusted", + trust: 'trusted', }).finish(); const back = Hello.decode(bytes); - expect(back.sandboxId).toBe("sbx-1"); - expect(back.labels.team).toBe("alpha"); - expect(back.capabilities).toEqual(["python3", "kubectl"]); + expect(back.sandboxId).toBe('sbx-1'); + expect(back.labels.team).toBe('alpha'); + expect(back.capabilities).toEqual(['python3', 'kubectl']); expect(back.capacityMax).toBe(4); }); - it("keeps req_id as a number and stdin as bytes", () => { + it('keeps req_id as a number and stdin as bytes', () => { const back = Exec.decode( Exec.encode({ reqId: 7, - command: "echo hi", + command: 'echo hi', stdin: new Uint8Array([1, 2, 3]), timeoutS: 30, streaming: true, @@ -36,21 +36,27 @@ describe("sandbox/v1 generated TypeScript stubs", () => { expect(Array.from(back.stdin)).toEqual([1, 2, 3]); }); - it("preserves a negative exit_code (sint32 zigzag)", () => { + it('preserves a negative exit_code (sint32 zigzag)', () => { const back = End.decode(End.encode({ reqId: 7, exitCode: -9 }).finish()); expect(back.exitCode).toBe(-9); }); }); -describe("Chunk stream discriminator (ST3)", () => { - it("exposes the Stream enum with stdout/stderr", () => { +describe('Chunk stream discriminator (ST3)', () => { + it('exposes the Stream enum with stdout/stderr', () => { expect(Stream.STREAM_STDOUT).toBe(1); expect(Stream.STREAM_STDERR).toBe(2); expect(Stream.STREAM_UNSPECIFIED).toBe(0); }); - it("round-trips a Chunk carrying stderr", () => { - const c = Chunk.decode(Chunk.encode({ reqId: 7, data: new Uint8Array([1, 2]), stream: Stream.STREAM_STDERR }).finish()); + it('round-trips a Chunk carrying stderr', () => { + const c = Chunk.decode( + Chunk.encode({ + reqId: 7, + data: new Uint8Array([1, 2]), + stream: Stream.STREAM_STDERR, + }).finish(), + ); expect(c.reqId).toBe(7); expect(c.stream).toBe(Stream.STREAM_STDERR); expect([...c.data]).toEqual([1, 2]); diff --git a/packages/k8s-sandbox/test/req-id.test.ts b/packages/k8s-sandbox/test/req-id.test.ts index 8d0b8f7..b2bc5a0 100644 --- a/packages/k8s-sandbox/test/req-id.test.ts +++ b/packages/k8s-sandbox/test/req-id.test.ts @@ -1,15 +1,17 @@ -import { describe, expect, it } from "vitest"; -import { makeReqIdSource } from "../src/req-id.js"; +import { describe, expect, it } from 'vitest'; +import { makeReqIdSource } from '../src/req-id.js'; -describe("makeReqIdSource", () => { - it("is monotonic within one source", () => { +describe('makeReqIdSource', () => { + it('is monotonic within one source', () => { const next = makeReqIdSource(); - const a = next(), b = next(), c = next(); + const a = next(), + b = next(), + c = next(); expect(b).toBeGreaterThan(a); expect(c).toBeGreaterThan(b); }); - it("yields disjoint id spaces across sources (the multi-replica property)", () => { + it('yields disjoint id spaces across sources (the multi-replica property)', () => { // Two independently-seeded sources stand in for two harness replicas sharing one // worker. This is the property that fixed #179: bare per-process counters both // emitted 1,2,3..., and because the relay keys its sinks by req_id alone, the @@ -17,13 +19,14 @@ describe("makeReqIdSource", () => { // is probabilistic, not guaranteed — for these TWO sources the flake rate is one // shared salt in 2^21 ≈ 4.8e-7. (The 4.8e-6 quoted in req-id.ts and ADR 0024 is // the five-replica birthday bound C(5,2)/2^21, a different number.) - const a = makeReqIdSource(), b = makeReqIdSource(); + const a = makeReqIdSource(), + b = makeReqIdSource(); const setA = new Set(Array.from({ length: 500 }, a)); const setB = new Set(Array.from({ length: 500 }, b)); expect([...setB].some((id) => setA.has(id))).toBe(false); }); - it("stays inside Number.MAX_SAFE_INTEGER for sampled ids; exhaustion boundary is checked by static arithmetic", () => { + it('stays inside Number.MAX_SAFE_INTEGER for sampled ids; exhaustion boundary is checked by static arithmetic', () => { // req_id is uint64 on the wire but a JS number after longToNumber, so an id past // 2^53-1 would silently lose precision and alias onto another exec. const next = makeReqIdSource(); diff --git a/packages/k8s-sandbox/test/resolve-pod.test.ts b/packages/k8s-sandbox/test/resolve-pod.test.ts index 706bef1..ae2472a 100644 --- a/packages/k8s-sandbox/test/resolve-pod.test.ts +++ b/packages/k8s-sandbox/test/resolve-pod.test.ts @@ -1,42 +1,102 @@ -import { describe, it, expect } from "vitest"; -import { buildSelectorArgs, buildPodNameArgs, resolveSandboxConfig } from "../src/resolve-pod.js"; +import { describe, it, expect } from 'vitest'; +import { buildSelectorArgs, buildPodNameArgs, resolveSandboxConfig } from '../src/resolve-pod.js'; -describe("arg builders", () => { - it("builds selector args with jsonpath and optional context", () => { - expect(buildSelectorArgs("sandbox-0", "default")).toEqual(["get", "sandbox", "sandbox-0", "-n", "default", "-o", "jsonpath={.status.selector}"]); - expect(buildSelectorArgs("s", "ns", "kind-x")).toEqual(["get", "sandbox", "s", "-n", "ns", "--context", "kind-x", "-o", "jsonpath={.status.selector}"]); +describe('arg builders', () => { + it('builds selector args with jsonpath and optional context', () => { + expect(buildSelectorArgs('sandbox-0', 'default')).toEqual([ + 'get', + 'sandbox', + 'sandbox-0', + '-n', + 'default', + '-o', + 'jsonpath={.status.selector}', + ]); + expect(buildSelectorArgs('s', 'ns', 'kind-x')).toEqual([ + 'get', + 'sandbox', + 's', + '-n', + 'ns', + '--context', + 'kind-x', + '-o', + 'jsonpath={.status.selector}', + ]); }); - it("builds pod-name args filtered to Running", () => { - expect(buildPodNameArgs("app=sandbox", "default")).toEqual(["get", "pod", "-n", "default", "-l", "app=sandbox", "--field-selector=status.phase=Running", "-o", "jsonpath={.items[0].metadata.name}"]); + it('builds pod-name args filtered to Running', () => { + expect(buildPodNameArgs('app=sandbox', 'default')).toEqual([ + 'get', + 'pod', + '-n', + 'default', + '-l', + 'app=sandbox', + '--field-selector=status.phase=Running', + '-o', + 'jsonpath={.items[0].metadata.name}', + ]); }); - it("builds pod-name args with context", () => { - expect(buildPodNameArgs("app=sandbox", "ns", "kind-x")).toEqual(["get", "pod", "-n", "ns", "-l", "app=sandbox", "--field-selector=status.phase=Running", "--context", "kind-x", "-o", "jsonpath={.items[0].metadata.name}"]); + it('builds pod-name args with context', () => { + expect(buildPodNameArgs('app=sandbox', 'ns', 'kind-x')).toEqual([ + 'get', + 'pod', + '-n', + 'ns', + '-l', + 'app=sandbox', + '--field-selector=status.phase=Running', + '--context', + 'kind-x', + '-o', + 'jsonpath={.items[0].metadata.name}', + ]); }); }); -describe("resolveSandboxConfig", () => { - it("short-circuits to KAGENTI_SANDBOX_POD without any kubectl call", async () => { +describe('resolveSandboxConfig', () => { + it('short-circuits to KAGENTI_SANDBOX_POD without any kubectl call', async () => { let called = false; - const cfg = await resolveSandboxConfig({ KAGENTI_SANDBOX_POD: "sbx-0" }, "/head", async () => { called = true; return ""; }); - expect(cfg?.pod).toBe("sbx-0"); + const cfg = await resolveSandboxConfig({ KAGENTI_SANDBOX_POD: 'sbx-0' }, '/head', async () => { + called = true; + return ''; + }); + expect(cfg?.pod).toBe('sbx-0'); expect(called).toBe(false); }); - it("returns null when neither POD nor NAME is set", async () => { - expect(await resolveSandboxConfig({}, "/head", async () => "")).toBeNull(); + it('returns null when neither POD nor NAME is set', async () => { + expect(await resolveSandboxConfig({}, '/head', async () => '')).toBeNull(); }); - it("resolves the pod via selector when NAME is set", async () => { + it('resolves the pod via selector when NAME is set', async () => { const calls: string[][] = []; - const run = async (args: string[]) => { calls.push(args); return args[1] === "sandbox" ? "app=sandbox,agents.x-k8s.io/sandbox=s" : "s-abc123"; }; - const cfg = await resolveSandboxConfig({ KAGENTI_SANDBOX_NAME: "s", KAGENTI_SANDBOX_NAMESPACE: "team1" }, "/head", run); - expect(cfg).toEqual({ pod: "s-abc123", namespace: "team1", context: undefined, podCwd: "/workspace", headCwd: "/head" }); - expect(calls[0]).toContain("sandbox"); - expect(calls[1]).toContain("-l"); - }); - it("throws when the Sandbox has no selector yet", async () => { - await expect(resolveSandboxConfig({ KAGENTI_SANDBOX_NAME: "s" }, "/head", async () => "")).rejects.toThrow(/selector/); - }); - it("throws when no Running pod matches the selector", async () => { - const run = async (args: string[]) => (args[1] === "sandbox" ? "app=sandbox" : ""); - await expect(resolveSandboxConfig({ KAGENTI_SANDBOX_NAME: "s" }, "/head", run)).rejects.toThrow(/no Running pod/); + const run = async (args: string[]) => { + calls.push(args); + return args[1] === 'sandbox' ? 'app=sandbox,agents.x-k8s.io/sandbox=s' : 's-abc123'; + }; + const cfg = await resolveSandboxConfig( + { KAGENTI_SANDBOX_NAME: 's', KAGENTI_SANDBOX_NAMESPACE: 'team1' }, + '/head', + run, + ); + expect(cfg).toEqual({ + pod: 's-abc123', + namespace: 'team1', + context: undefined, + podCwd: '/workspace', + headCwd: '/head', + }); + expect(calls[0]).toContain('sandbox'); + expect(calls[1]).toContain('-l'); + }); + it('throws when the Sandbox has no selector yet', async () => { + await expect( + resolveSandboxConfig({ KAGENTI_SANDBOX_NAME: 's' }, '/head', async () => ''), + ).rejects.toThrow(/selector/); + }); + it('throws when no Running pod matches the selector', async () => { + const run = async (args: string[]) => (args[1] === 'sandbox' ? 'app=sandbox' : ''); + await expect(resolveSandboxConfig({ KAGENTI_SANDBOX_NAME: 's' }, '/head', run)).rejects.toThrow( + /no Running pod/, + ); }); }); diff --git a/packages/k8s-sandbox/test/transport-conformance.test.ts b/packages/k8s-sandbox/test/transport-conformance.test.ts index 1632d06..e0efed8 100644 --- a/packages/k8s-sandbox/test/transport-conformance.test.ts +++ b/packages/k8s-sandbox/test/transport-conformance.test.ts @@ -1,17 +1,17 @@ -import { EventEmitter } from "node:events"; -import { vi } from "vitest"; -import type { K8sSandboxConfig } from "../src/config.js"; -import { KubectlTransport } from "../src/exec.js"; -import { runConformance, type TransportFactory } from "./conformance.js"; +import { EventEmitter } from 'node:events'; +import { vi } from 'vitest'; +import type { K8sSandboxConfig } from '../src/config.js'; +import { KubectlTransport } from '../src/exec.js'; +import { runConformance, type TransportFactory } from './conformance.js'; -type SpawnFn = typeof import("node:child_process").spawn; +type SpawnFn = typeof import('node:child_process').spawn; const cfg: K8sSandboxConfig = { - pod: "sbx-0", - namespace: "team1", + pod: 'sbx-0', + namespace: 'team1', context: undefined, - podCwd: "/workspace", - headCwd: "/head", + podCwd: '/workspace', + headCwd: '/head', }; /** Build a KubectlTransport whose child process is a scripted fake. */ @@ -25,18 +25,22 @@ const kubectlFactory: TransportFactory = (b, opts) => { const child = new EventEmitter() as any; child.stdout = new EventEmitter(); child.stderr = new EventEmitter(); - child.stdin = { end: (d?: Buffer) => { stdin = d; } }; + child.stdin = { + end: (d?: Buffer) => { + stdin = d; + }, + }; // kill() drives a `close` event, exactly as a real SIGKILL would. child.kill = vi.fn(() => { killed = true; - child.emit("close", null); + child.emit('close', null); }); // Emit after the transport has attached its handlers (still synchronous // relative to the awaiting test via a microtask). queueMicrotask(() => { - for (const s of b.stdout ?? []) child.stdout.emit("data", Buffer.from(s)); - for (const s of b.stderr ?? []) child.stderr.emit("data", Buffer.from(s)); - if (!b.hang) child.emit("close", b.exitCode ?? 0); + for (const s of b.stdout ?? []) child.stdout.emit('data', Buffer.from(s)); + for (const s of b.stderr ?? []) child.stderr.emit('data', Buffer.from(s)); + if (!b.hang) child.emit('close', b.exitCode ?? 0); }); return child; }) as unknown as SpawnFn; @@ -44,7 +48,11 @@ const kubectlFactory: TransportFactory = (b, opts) => { // KubectlTransport can only kill its own `kubectl exec` client; the in-pod process is // then stopped, if at all, by EPIPE on its next write. `killed` is the only witness // that the kill happened at all — the fake emits `close` on its own. - return { transport, stdinSeen: () => stdin, producerStop: () => (killed ? "local-kill" : "none") }; + return { + transport, + stdinSeen: () => stdin, + producerStop: () => (killed ? 'local-kill' : 'none'), + }; }; -runConformance("KubectlTransport", kubectlFactory, { producerStop: "local-kill", streams: true }); +runConformance('KubectlTransport', kubectlFactory, { producerStop: 'local-kill', streams: true }); diff --git a/packages/knative-server/src/context-service.ts b/packages/knative-server/src/context-service.ts index 7203738..063feb1 100644 --- a/packages/knative-server/src/context-service.ts +++ b/packages/knative-server/src/context-service.ts @@ -31,7 +31,7 @@ interface ContextPool { replicas: number; readyReplicas: number; sandboxSelector: string; - workspace: WorkloadRecord["workspace"]; + workspace: WorkloadRecord['workspace']; } export function contextServiceConfigured(): boolean { @@ -40,19 +40,20 @@ export function contextServiceConfigured(): boolean { function baseUrl(): string { const configured = process.env.CONTEXT_SERVICE_URL?.trim(); - if (!configured) throw new Error("Context Service is not configured"); - return configured.replace(/\/$/, ""); + if (!configured) throw new Error('Context Service is not configured'); + return configured.replace(/\/$/, ''); } async function request(path: string, init?: RequestInit): Promise { - const configuredTimeout = Number.parseInt(process.env.CONTEXT_SERVICE_TIMEOUT_MS ?? "5000", 10); - const timeoutMs = Number.isFinite(configuredTimeout) && configuredTimeout > 0 ? configuredTimeout : 5000; + const configuredTimeout = Number.parseInt(process.env.CONTEXT_SERVICE_TIMEOUT_MS ?? '5000', 10); + const timeoutMs = + Number.isFinite(configuredTimeout) && configuredTimeout > 0 ? configuredTimeout : 5000; const controller = new AbortController(); const timeout = setTimeout(() => controller.abort(), timeoutMs); try { const response = await fetch(`${baseUrl()}${path}`, { ...init, signal: controller.signal }); if (response.ok) return response; - const body = await response.json().catch(() => ({})) as { message?: string }; + const body = (await response.json().catch(() => ({}))) as { message?: string }; throw new Error(body.message ?? `Context Service returned ${response.status}`); } finally { clearTimeout(timeout); @@ -70,33 +71,36 @@ function workload(pool: ContextPool): WorkloadRecord { }; } -export async function createWorkload(workloadId: string, spec: WorkloadRequest): Promise { +export async function createWorkload( + workloadId: string, + spec: WorkloadRequest, +): Promise { const shared = spec.workspace?.shared === true; const claimName = spec.workspace?.claimName; const workspace = claimName ? { claimName, readOnly: spec.workspace?.readOnly } : { - size: spec.workspace?.size ?? "1Gi", - accessMode: shared ? "ReadWriteMany" : "ReadWriteOnce", + size: spec.workspace?.size ?? '1Gi', + accessMode: shared ? 'ReadWriteMany' : 'ReadWriteOnce', ...(spec.workspace?.storageClass ? { storageClass: spec.workspace.storageClass } : {}), }; - const response = await request("/v1/sandbox-pools", { - method: "POST", - headers: { "content-type": "application/json" }, + const response = await request('/v1/sandbox-pools', { + method: 'POST', + headers: { 'content-type': 'application/json' }, body: JSON.stringify({ name: workloadId, replicas: spec.sandboxes ?? (shared ? 2 : 1), workspace, }), }); - return workload(await response.json() as ContextPool); + return workload((await response.json()) as ContextPool); } export async function getWorkload(workloadId: string): Promise { const response = await request(`/v1/sandbox-pools/${encodeURIComponent(workloadId)}`); - return workload(await response.json() as ContextPool); + return workload((await response.json()) as ContextPool); } export async function deleteWorkload(workloadId: string): Promise { - await request(`/v1/sandbox-pools/${encodeURIComponent(workloadId)}`, { method: "DELETE" }); + await request(`/v1/sandbox-pools/${encodeURIComponent(workloadId)}`, { method: 'DELETE' }); } diff --git a/packages/knative-server/src/cron-dispatch.ts b/packages/knative-server/src/cron-dispatch.ts index 53b66bf..0e21ce8 100644 --- a/packages/knative-server/src/cron-dispatch.ts +++ b/packages/knative-server/src/cron-dispatch.ts @@ -1,11 +1,14 @@ -import { readFileSync } from "node:fs"; -import { fileURLToPath } from "node:url"; +import { readFileSync } from 'node:fs'; +import { fileURLToPath } from 'node:url'; /** Replace every __FIRE__ in each string field with fireId; non-strings pass through. Pure, non-mutating. */ -export function applyFire(envelope: Record, fireId: string): Record { +export function applyFire( + envelope: Record, + fireId: string, +): Record { const out: Record = {}; for (const [k, v] of Object.entries(envelope)) { - out[k] = typeof v === "string" ? v.split("__FIRE__").join(fireId) : v; + out[k] = typeof v === 'string' ? v.split('__FIRE__').join(fireId) : v; } return out; } @@ -38,7 +41,7 @@ export function exitCodeFor(result: { failed: number }): number { } export function loadConfig(path: string): Record[] { - const parsed = JSON.parse(readFileSync(path, "utf8")); + const parsed = JSON.parse(readFileSync(path, 'utf8')); if (!Array.isArray(parsed?.items)) throw new Error("cron config: 'items' must be an array"); return parsed.items as Record[]; } @@ -50,11 +53,11 @@ export function loadConfig(path: string): Record[] { * passes whatever the config provides through unchanged (after __FIRE__ substitution). */ function buildPost(): (env: Record) => Promise { - const base = process.env.SH_SERVICE_URL ?? "http://serverless-harness.default.svc.cluster.local"; + const base = process.env.SH_SERVICE_URL ?? 'http://serverless-harness.default.svc.cluster.local'; return async (env) => { const res = await fetch(`${base}/runs`, { - method: "POST", - headers: { "content-type": "application/json" }, + method: 'POST', + headers: { 'content-type': 'application/json' }, body: JSON.stringify(env), }); if (res.status !== 202) { @@ -62,17 +65,20 @@ function buildPost(): (env: Record) => Promise { return false; } const body = await res.json().catch(() => ({}) as Record); - return (body as Record).status === "accepted"; + return (body as Record).status === 'accepted'; }; } async function main(): Promise { const fireId = process.env.JOB_NAME ?? `manual-${process.pid}`; - const configPath = process.env.CRON_CONFIG ?? "/config/schedule.json"; + const configPath = process.env.CRON_CONFIG ?? '/config/schedule.json'; const items = loadConfig(configPath); - if (items.length === 0) console.warn(`cron-dispatch: config has no items — nothing to dispatch (fire=${fireId})`); + if (items.length === 0) + console.warn(`cron-dispatch: config has no items — nothing to dispatch (fire=${fireId})`); const result = await dispatchAll(items, fireId, buildPost()); - console.log(`cron-dispatch: ${result.accepted}/${result.total} accepted, ${result.failed} failed (fire=${fireId})`); + console.log( + `cron-dispatch: ${result.accepted}/${result.total} accepted, ${result.failed} failed (fire=${fireId})`, + ); process.exit(exitCodeFor(result)); } @@ -80,7 +86,7 @@ async function main(): Promise { const isMainModule = process.argv[1] === fileURLToPath(import.meta.url); if (isMainModule) { main().catch((err) => { - console.error("cron-dispatch error:", err); + console.error('cron-dispatch error:', err); process.exit(1); }); } diff --git a/packages/knative-server/src/index.ts b/packages/knative-server/src/index.ts index ed26b1a..bfc57c7 100644 --- a/packages/knative-server/src/index.ts +++ b/packages/knative-server/src/index.ts @@ -1 +1 @@ -export { startServer } from "./server.js"; +export { startServer } from './server.js'; diff --git a/packages/knative-server/src/leaf-job.ts b/packages/knative-server/src/leaf-job.ts index 00bd9f3..5b5218f 100644 --- a/packages/knative-server/src/leaf-job.ts +++ b/packages/knative-server/src/leaf-job.ts @@ -1,14 +1,14 @@ // packages/knative-server/src/leaf-job.ts -import { RedisWorkQueue } from "@sh/work-queue"; -import { processOne } from "@sh/harness/leaf-job-runner"; -import { runLeaf, leafSessionId, type LeafEnvelope } from "@sh/harness/run-leaf"; -import { RedisResultStore, toResultRecord, writeResult } from "@sh/harness/leaf-result-store"; -import { type TurnConfig } from "@sh/harness/run-turn"; +import { RedisWorkQueue } from '@sh/work-queue'; +import { processOne } from '@sh/harness/leaf-job-runner'; +import { runLeaf, leafSessionId, type LeafEnvelope } from '@sh/harness/run-leaf'; +import { RedisResultStore, toResultRecord, writeResult } from '@sh/harness/leaf-result-store'; +import { type TurnConfig } from '@sh/harness/run-turn'; const MIN_IDLE_MS = 90_000; const MAX_ATTEMPTS = 3; const CONSUMER_GC_IDLE_MS = 300_000; // 5 min — GC consumers idle longer than this with 0 pending -const RESULT_TTL_SECONDS = parseInt(process.env.LEAF_RESULT_TTL_SECONDS ?? "86400", 10); +const RESULT_TTL_SECONDS = parseInt(process.env.LEAF_RESULT_TTL_SECONDS ?? '86400', 10); function buildConfig(): TurnConfig { return { @@ -32,14 +32,27 @@ async function main(): Promise { // Startup reap: dead-letter stale PEL entries past maxAttempts so pendingEntriesCount drops // and KEDA can scale to zero without waiting for reclaim cycles. - const deadLettered = await q.reapDeadLetters(consumerId, { minIdleMs: MIN_IDLE_MS, maxAttempts: MAX_ATTEMPTS }); + const deadLettered = await q.reapDeadLetters(consumerId, { + minIdleMs: MIN_IDLE_MS, + maxAttempts: MAX_ATTEMPTS, + }); for (const { entryId, envelope } of deadLettered) { - if (envelope && typeof envelope === "object") { + if (envelope && typeof envelope === 'object') { const env = envelope as LeafEnvelope; try { - await writeResult(resultStore, leafSessionId(env), - toResultRecord({ status: "failed", reason: "error" }, env.sessionId, new Date().toISOString()), RESULT_TTL_SECONDS); - } catch { /* best-effort record write */ } + await writeResult( + resultStore, + leafSessionId(env), + toResultRecord( + { status: 'failed', reason: 'error' }, + env.sessionId, + new Date().toISOString(), + ), + RESULT_TTL_SECONDS, + ); + } catch { + /* best-effort record write */ + } } else { console.warn(`reaper: entry ${entryId} dead-lettered with unrecoverable envelope`); } @@ -66,8 +79,8 @@ async function main(): Promise { ttlSeconds: RESULT_TTL_SECONDS, consumerId, }); - if (outcome === "idle") break; - if (outcome === "retry") { + if (outcome === 'idle') break; + if (outcome === 'retry') { // Entry stays pending for another pod to reclaim — do NOT delete our consumer. await q.close(); await resultStore.close(); @@ -83,7 +96,7 @@ async function main(): Promise { } main().catch(async (err) => { - console.error("leaf-job error:", err); + console.error('leaf-job error:', err); try { await queue?.close(); } catch { diff --git a/packages/knative-server/src/server.ts b/packages/knative-server/src/server.ts index 89fb4d9..743e545 100644 --- a/packages/knative-server/src/server.ts +++ b/packages/knative-server/src/server.ts @@ -1,11 +1,22 @@ -import { createServer, type IncomingMessage, type ServerResponse } from "node:http"; -import { randomUUID } from "node:crypto"; -import { fileURLToPath } from "node:url"; -import { runTurn, executeTurn, type TurnConfig } from "@sh/harness/run-turn"; -import { terminalFrame, type TurnStreamFrame } from "@sh/harness/turn-stream"; -import { runLeaf, leafSessionId, validateItem, type LeafEnvelope, type LeafResult } from "@sh/harness/run-leaf"; -import { RedisWorkQueue } from "@sh/work-queue"; -import { RedisResultStore, toResultRecord, writeResult, readResult } from "@sh/harness/leaf-result-store"; +import { createServer, type IncomingMessage, type ServerResponse } from 'node:http'; +import { randomUUID } from 'node:crypto'; +import { fileURLToPath } from 'node:url'; +import { runTurn, executeTurn, type TurnConfig } from '@sh/harness/run-turn'; +import { terminalFrame, type TurnStreamFrame } from '@sh/harness/turn-stream'; +import { + runLeaf, + leafSessionId, + validateItem, + type LeafEnvelope, + type LeafResult, +} from '@sh/harness/run-leaf'; +import { RedisWorkQueue } from '@sh/work-queue'; +import { + RedisResultStore, + toResultRecord, + writeResult, + readResult, +} from '@sh/harness/leaf-result-store'; import { contextServiceConfigured, createWorkload, @@ -13,17 +24,17 @@ import { getWorkload, type WorkloadRecord, type WorkloadRequest, -} from "./context-service.js"; +} from './context-service.js'; -const PORT = parseInt(process.env.PORT || "8080", 10); -const JSON_HEADERS = { "Content-Type": "application/json" }; +const PORT = parseInt(process.env.PORT || '8080', 10); +const JSON_HEADERS = { 'Content-Type': 'application/json' }; const SSE_HEADERS = { - "Content-Type": "text/event-stream", - "Cache-Control": "no-cache", - Connection: "keep-alive", - "X-Accel-Buffering": "no", // belt-and-suspenders for any nginx fronting Kourier (Envoy ignores it) + 'Content-Type': 'text/event-stream', + 'Cache-Control': 'no-cache', + Connection: 'keep-alive', + 'X-Accel-Buffering': 'no', // belt-and-suspenders for any nginx fronting Kourier (Envoy ignores it) }; -const RESULT_TTL_SECONDS = parseInt(process.env.LEAF_RESULT_TTL_SECONDS ?? "86400", 10); +const RESULT_TTL_SECONDS = parseInt(process.env.LEAF_RESULT_TTL_SECONDS ?? '86400', 10); const WORKLOAD_NAME = /^[a-z0-9](?:[-a-z0-9]*[a-z0-9])?$/; const sleep = (ms: number) => new Promise((resolve) => setTimeout(resolve, ms)); @@ -43,14 +54,14 @@ function intEnv(name: string, def: number): number { // request so overrides take effect without a restart (and so tests can shrink the budget). function saturationWaitConfig() { return { - waitMs: intEnv("KAGENTI_SYNC_SATURATION_WAIT_MS", 30000), - backoffMs: intEnv("KAGENTI_SYNC_SATURATION_BACKOFF_MS", 250), - maxBackoffMs: intEnv("KAGENTI_SYNC_SATURATION_MAX_BACKOFF_MS", 5000), - retryAfterS: intEnv("KAGENTI_SYNC_SATURATION_RETRY_AFTER_S", 5), + waitMs: intEnv('KAGENTI_SYNC_SATURATION_WAIT_MS', 30000), + backoffMs: intEnv('KAGENTI_SYNC_SATURATION_BACKOFF_MS', 250), + maxBackoffMs: intEnv('KAGENTI_SYNC_SATURATION_MAX_BACKOFF_MS', 5000), + retryAfterS: intEnv('KAGENTI_SYNC_SATURATION_RETRY_AFTER_S', 5), }; } -const isSaturated = (r: LeafResult): boolean => r.status === "failed" && r.reason === "saturated"; +const isSaturated = (r: LeafResult): boolean => r.status === 'failed' && r.reason === 'saturated'; function buildConfig(): TurnConfig { return { @@ -64,9 +75,9 @@ function buildConfig(): TurnConfig { function readBody(req: IncomingMessage): Promise { return new Promise((resolve, reject) => { const chunks: Buffer[] = []; - req.on("data", (chunk: Buffer) => chunks.push(chunk)); - req.on("end", () => resolve(Buffer.concat(chunks).toString())); - req.on("error", reject); + req.on('data', (chunk: Buffer) => chunks.push(chunk)); + req.on('end', () => resolve(Buffer.concat(chunks).toString())); + req.on('error', reject); }); } @@ -75,7 +86,7 @@ async function handleTurn(req: IncomingMessage, res: ServerResponse): Promise { - if (!res.writableEnded) res.write(": keepalive\n\n"); // SSE comment — invisible to EventSource + if (!res.writableEnded) res.write(': keepalive\n\n'); // SSE comment — invisible to EventSource }, keepaliveMs); }; const writeFrame = (frame: TurnStreamFrame) => { @@ -148,14 +159,14 @@ async function handleTurnStream( // normal completion (res.end() already called) is a no-op; only a premature close aborts. On // Node 22 the reliable disconnect signal for a half-consumed streaming request is the RESPONSE's // 'close' (the request's own 'close' fires with request-body end, not on socket teardown here). - res.on("close", () => { + res.on('close', () => { if (!res.writableEnded) { clientGone = true; ac.abort(); } }); - const { writeFrame, stop } = makeFrameWriter(res, intEnv("SH_TURN_STREAM_KEEPALIVE_MS", 20000)); + const { writeFrame, stop } = makeFrameWriter(res, intEnv('SH_TURN_STREAM_KEEPALIVE_MS', 20000)); try { const result = await executeTurn({ prompt, @@ -173,10 +184,10 @@ async function handleTurnStream( // Pre-first-frame: nothing streamed yet, so reuse the EXACT sync mapping — a bad sessionId // still returns real 404 JSON, byte-identical to the sync path (§3.4 regime 2). const message = err instanceof Error ? err.message : String(err); - const status = message.includes("no session in backend") ? 404 : 500; + const status = message.includes('no session in backend') ? 404 : 500; res.writeHead(status, JSON_HEADERS).end( JSON.stringify({ - error: status === 404 ? "session_not_found" : message, + error: status === 404 ? 'session_not_found' : message, ...(sessionId ? { sessionId } : {}), }), ); @@ -189,9 +200,9 @@ async function handleTurnStream( const message = err instanceof Error ? err.message : String(err); res.write( `event: error\ndata: ${JSON.stringify({ - type: "error", - sessionId: sessionId ?? "", - stopReason: "error", + type: 'error', + sessionId: sessionId ?? '', + stopReason: 'error', errorMessage: message, })}\n\n`, ); @@ -203,18 +214,24 @@ async function handleTurnStream( } export function isLeafEnvelope(o: any): o is LeafEnvelope { - return o && typeof o.sessionId === "string" && validateItem(o.item) !== null; + return o && typeof o.sessionId === 'string' && validateItem(o.item) !== null; } export function isSolveEnvelope(o: any): boolean { - return o && typeof o.sessionId === "string" && o.kind === "solve" - && typeof o.problemStatement === "string" - && typeof o.repoUrl === "string" && typeof o.ref === "string"; + return ( + o && + typeof o.sessionId === 'string' && + o.kind === 'solve' && + typeof o.problemStatement === 'string' && + typeof o.repoUrl === 'string' && + typeof o.ref === 'string' + ); } export function isPromptEnvelope(o: any): boolean { - return o && typeof o.sessionId === "string" && o.kind === "prompt" - && typeof o.prompt === "string"; + return ( + o && typeof o.sessionId === 'string' && o.kind === 'prompt' && typeof o.prompt === 'string' + ); } export function isRunEnvelope(o: any): boolean { @@ -242,28 +259,32 @@ async function saveWorkload(record: WorkloadRecord): Promise { async function findWorkload(id: string): Promise { const raw = await getResultStore().get(workloadKey(id)); if (!raw) return null; - try { return JSON.parse(raw) as WorkloadRecord; } catch { return null; } + try { + return JSON.parse(raw) as WorkloadRecord; + } catch { + return null; + } } function requireContextService(res: ServerResponse): boolean { if (contextServiceConfigured()) return true; - res.writeHead(501, JSON_HEADERS).end(JSON.stringify({ error: "context_service_not_configured" })); + res.writeHead(501, JSON_HEADERS).end(JSON.stringify({ error: 'context_service_not_configured' })); return false; } function contextServiceFailure(operation: string, err: unknown, res: ServerResponse): void { console.error(`Context Service ${operation} failed:`, err); - res.writeHead(502, JSON_HEADERS).end(JSON.stringify({ error: "context_service_error" })); + res.writeHead(502, JSON_HEADERS).end(JSON.stringify({ error: 'context_service_error' })); } async function resolveRunWorkload(body: any, res: ServerResponse): Promise { if (!body?.workloadId) return body; const record = await findWorkload(body.workloadId); - if (!record || record.status === "deleted") { - res.writeHead(404, JSON_HEADERS).end(JSON.stringify({ error: "workload_not_found" })); + if (!record || record.status === 'deleted') { + res.writeHead(404, JSON_HEADERS).end(JSON.stringify({ error: 'workload_not_found' })); return null; } - if (body.kind === "prompt") { + if (body.kind === 'prompt') { // A prompt leaf DOES lease a pool sandbox now, and honors an envelope `sandboxPoolSelector` // (ADR 0028 amendment, 2026-09-01) — but a *workload-addressed* one still ignores the // workload's own selector. The workloadId gates existence (404 above) and nothing more. @@ -272,7 +293,9 @@ async function resolveRunWorkload(body: any, res: ServerResponse): Promise { if (!requireContextService(res)) return; let spec: WorkloadRequest; - try { spec = JSON.parse(await readBody(req)) as WorkloadRequest; } - catch { res.writeHead(400, JSON_HEADERS).end(JSON.stringify({ error: "invalid_json" })); return; } + try { + spec = JSON.parse(await readBody(req)) as WorkloadRequest; + } catch { + res.writeHead(400, JSON_HEADERS).end(JSON.stringify({ error: 'invalid_json' })); + return; + } const workloadId = spec.name ?? `wl-${randomUUID().slice(0, 8)}`; if (workloadId.length > 50 || !WORKLOAD_NAME.test(workloadId)) { - res.writeHead(400, JSON_HEADERS).end(JSON.stringify({ error: "workload_name_invalid" })); + res.writeHead(400, JSON_HEADERS).end(JSON.stringify({ error: 'workload_name_invalid' })); return; } try { @@ -294,14 +321,14 @@ async function handleCreateWorkload(req: IncomingMessage, res: ServerResponse): await saveWorkload(record); res.writeHead(201, JSON_HEADERS).end(JSON.stringify(record)); } catch (err) { - contextServiceFailure("create", err, res); + contextServiceFailure('create', err, res); } } async function handleGetWorkload(id: string, res: ServerResponse): Promise { if (!requireContextService(res)) return; - if (!await findWorkload(id)) { - res.writeHead(404, JSON_HEADERS).end(JSON.stringify({ error: "workload_not_found" })); + if (!(await findWorkload(id))) { + res.writeHead(404, JSON_HEADERS).end(JSON.stringify({ error: 'workload_not_found' })); return; } try { @@ -309,38 +336,46 @@ async function handleGetWorkload(id: string, res: ServerResponse): Promise await saveWorkload(record); res.writeHead(200, JSON_HEADERS).end(JSON.stringify(record)); } catch (err) { - contextServiceFailure("get", err, res); + contextServiceFailure('get', err, res); } } async function handleDeleteWorkload(id: string, res: ServerResponse): Promise { if (!requireContextService(res)) return; const record = await findWorkload(id); - if (!record || record.status === "deleted") { - res.writeHead(404, JSON_HEADERS).end(JSON.stringify({ error: "workload_not_found" })); + if (!record || record.status === 'deleted') { + res.writeHead(404, JSON_HEADERS).end(JSON.stringify({ error: 'workload_not_found' })); return; } try { await deleteWorkload(id); - await saveWorkload({ ...record, status: "deleted", readyReplicas: 0 }); + await saveWorkload({ ...record, status: 'deleted', readyReplicas: 0 }); res.writeHead(204).end(); } catch (err) { - contextServiceFailure("delete", err, res); + contextServiceFailure('delete', err, res); } } async function handleEnqueueLeafParsed(body: any, res: ServerResponse): Promise { - if (!isRunEnvelope(body)) { res.writeHead(400, JSON_HEADERS).end(JSON.stringify({ error: "envelope_invalid" })); return; } + if (!isRunEnvelope(body)) { + res.writeHead(400, JSON_HEADERS).end(JSON.stringify({ error: 'envelope_invalid' })); + return; + } body = await resolveRunWorkload(body, res); if (!body) return; const q = getQueue(); await q.ensureGroup(); await q.enqueue(body); - res.writeHead(202, JSON_HEADERS).end(JSON.stringify({ status: "accepted", sessionId: body.sessionId })); + res + .writeHead(202, JSON_HEADERS) + .end(JSON.stringify({ status: 'accepted', sessionId: body.sessionId })); } async function handleRunLeafParsed(body: any, _raw: string, res: ServerResponse): Promise { - if (!isRunEnvelope(body)) { res.writeHead(400, JSON_HEADERS).end(JSON.stringify({ error: "envelope_invalid" })); return; } + if (!isRunEnvelope(body)) { + res.writeHead(400, JSON_HEADERS).end(JSON.stringify({ error: 'envelope_invalid' })); + return; + } body = await resolveRunWorkload(body, res); if (!body) return; @@ -361,26 +396,61 @@ async function handleRunLeafParsed(body: any, _raw: string, res: ServerResponse) if (isSaturated(result)) { // Still saturated after the budget: tell the client to retry. Do NOT persist a result record — // a 503 is "retry", not a terminal failure, and /runs/status must not report it as one. - res.writeHead(503, { ...JSON_HEADERS, "Retry-After": String(cfg.retryAfterS) }) - .end(JSON.stringify({ status: "failed", reason: "saturated" })); + res + .writeHead(503, { ...JSON_HEADERS, 'Retry-After': String(cfg.retryAfterS) }) + .end(JSON.stringify({ status: 'failed', reason: 'saturated' })); return; } - await writeResult(getResultStore(), leafSessionId(body), toResultRecord(result, body.sessionId, new Date().toISOString()), RESULT_TTL_SECONDS); + await writeResult( + getResultStore(), + leafSessionId(body), + toResultRecord(result, body.sessionId, new Date().toISOString()), + RESULT_TTL_SECONDS, + ); res.writeHead(200, JSON_HEADERS).end(JSON.stringify(result)); } async function handleLeafStatus(url: URL, res: ServerResponse): Promise { - const sessionId = url.searchParams.get("sessionId"); - if (!sessionId) { res.writeHead(400, JSON_HEADERS).end(JSON.stringify({ error: "sessionId_required" })); return; } - const tenant = url.searchParams.get("tenant") ?? undefined; + const sessionId = url.searchParams.get('sessionId'); + if (!sessionId) { + res.writeHead(400, JSON_HEADERS).end(JSON.stringify({ error: 'sessionId_required' })); + return; + } + const tenant = url.searchParams.get('tenant') ?? undefined; const record = await readResult(getResultStore(), leafSessionId({ sessionId, tenant })); - if (!record) { res.writeHead(200, JSON_HEADERS).end(JSON.stringify({ status: "queued" })); return; } - if (record.status === "done") { res.writeHead(200, JSON_HEADERS).end(JSON.stringify({ status: "done", verdict: record.verdict })); return; } - if (record.status === "solved") { res.writeHead(200, JSON_HEADERS).end(JSON.stringify({ status: "solved", patch: record.patch })); return; } - if (record.status === "paused") { res.writeHead(200, JSON_HEADERS).end(JSON.stringify({ status: "paused", gateId: record.gate?.gateId, gate: record.gate })); return; } - if (record.status === "failed") { res.writeHead(200, JSON_HEADERS).end(JSON.stringify({ status: "failed", reason: record.reason ?? undefined })); return; } - if (record.status === "responded") { res.writeHead(200, JSON_HEADERS).end(JSON.stringify({ status: "responded", text: record.text })); return; } + if (!record) { + res.writeHead(200, JSON_HEADERS).end(JSON.stringify({ status: 'queued' })); + return; + } + if (record.status === 'done') { + res + .writeHead(200, JSON_HEADERS) + .end(JSON.stringify({ status: 'done', verdict: record.verdict })); + return; + } + if (record.status === 'solved') { + res.writeHead(200, JSON_HEADERS).end(JSON.stringify({ status: 'solved', patch: record.patch })); + return; + } + if (record.status === 'paused') { + res + .writeHead(200, JSON_HEADERS) + .end(JSON.stringify({ status: 'paused', gateId: record.gate?.gateId, gate: record.gate })); + return; + } + if (record.status === 'failed') { + res + .writeHead(200, JSON_HEADERS) + .end(JSON.stringify({ status: 'failed', reason: record.reason ?? undefined })); + return; + } + if (record.status === 'responded') { + res + .writeHead(200, JSON_HEADERS) + .end(JSON.stringify({ status: 'responded', text: record.text })); + return; + } res.writeHead(200, JSON_HEADERS).end(JSON.stringify({ status: record.status })); } @@ -388,8 +458,8 @@ async function handleLeafStatus(url: URL, res: ServerResponse): Promise { // industry-standard "run" noun (`/runs`); the internal `runLeaf`/`LeafEnvelope` vocabulary is // unchanged. Aliases warn once per path and are removed in a later release. const DEPRECATED_ROUTE_ALIASES: Record = { - "/run-leaf": "/runs", - "/run-leaf/status": "/runs/status", + '/run-leaf': '/runs', + '/run-leaf/status': '/runs/status', }; const warnedDeprecatedRoutes = new Set(); function warnDeprecatedRoute(oldPath: string): void { @@ -401,66 +471,78 @@ function warnDeprecatedRoute(oldPath: string): void { } function handler(req: IncomingMessage, res: ServerResponse): void { - const url = req.url ?? ""; + const url = req.url ?? ''; - if (req.method === "GET" && url === "/health") { - res.writeHead(200).end("ok"); + if (req.method === 'GET' && url === '/health') { + res.writeHead(200).end('ok'); return; } - if (req.method === "POST" && url === "/workloads") { + if (req.method === 'POST' && url === '/workloads') { handleCreateWorkload(req, res).catch((err) => { - if (!res.headersSent) res.writeHead(500, JSON_HEADERS).end(JSON.stringify({ error: String(err) })); + if (!res.headersSent) + res.writeHead(500, JSON_HEADERS).end(JSON.stringify({ error: String(err) })); }); return; } const workloadMatch = url.match(/^\/workloads\/([^/?]+)$/); - if (workloadMatch && req.method === "GET") { + if (workloadMatch && req.method === 'GET') { handleGetWorkload(decodeURIComponent(workloadMatch[1]), res).catch((err) => { - if (!res.headersSent) res.writeHead(500, JSON_HEADERS).end(JSON.stringify({ error: String(err) })); + if (!res.headersSent) + res.writeHead(500, JSON_HEADERS).end(JSON.stringify({ error: String(err) })); }); return; } - if (workloadMatch && req.method === "DELETE") { + if (workloadMatch && req.method === 'DELETE') { handleDeleteWorkload(decodeURIComponent(workloadMatch[1]), res).catch((err) => { - if (!res.headersSent) res.writeHead(500, JSON_HEADERS).end(JSON.stringify({ error: String(err) })); + if (!res.headersSent) + res.writeHead(500, JSON_HEADERS).end(JSON.stringify({ error: String(err) })); }); return; } // Run-status endpoint: canonical `/runs/status`, plus the deprecated `/run-leaf/status` alias. - if (req.method === "GET" && (url.startsWith("/runs/status") || url.startsWith("/run-leaf/status"))) { - if (url.startsWith("/run-leaf/status")) warnDeprecatedRoute("/run-leaf/status"); - handleLeafStatus(new URL(url, "http://localhost"), res).catch((err) => { - if (!res.headersSent) res.writeHead(500, JSON_HEADERS).end(JSON.stringify({ error: String(err) })); + if ( + req.method === 'GET' && + (url.startsWith('/runs/status') || url.startsWith('/run-leaf/status')) + ) { + if (url.startsWith('/run-leaf/status')) warnDeprecatedRoute('/run-leaf/status'); + handleLeafStatus(new URL(url, 'http://localhost'), res).catch((err) => { + if (!res.headersSent) + res.writeHead(500, JSON_HEADERS).end(JSON.stringify({ error: String(err) })); }); return; } // Run endpoint: canonical `POST /runs`, plus the deprecated `POST /run-leaf` alias. - if (req.method === "POST" && (url === "/runs" || url === "/run-leaf")) { - if (url === "/run-leaf") warnDeprecatedRoute("/run-leaf"); + if (req.method === 'POST' && (url === '/runs' || url === '/run-leaf')) { + if (url === '/run-leaf') warnDeprecatedRoute('/run-leaf'); const route = async () => { const raw = await readBody(req); let parsed: any = {}; - try { parsed = JSON.parse(raw); } catch { /* handled below */ } + try { + parsed = JSON.parse(raw); + } catch { + /* handled below */ + } // Pool selection is internal routing state. Never accept a Kubernetes selector directly // from an external run request; a workload resolver may add one after this boundary. - if (parsed && typeof parsed === "object") delete parsed.sandboxPoolSelector; + if (parsed && typeof parsed === 'object') delete parsed.sandboxPoolSelector; if (parsed && parsed.async === true) return handleEnqueueLeafParsed(parsed, res); return handleRunLeafParsed(parsed, raw, res); }; - route().catch((err) => { if (!res.headersSent) res.writeHead(500, JSON_HEADERS).end(JSON.stringify({ error: String(err) })); }); + route().catch((err) => { + if (!res.headersSent) + res.writeHead(500, JSON_HEADERS).end(JSON.stringify({ error: String(err) })); + }); return; } - if (req.method === "POST" && req.url === "/turn") { + if (req.method === 'POST' && req.url === '/turn') { handleTurn(req, res).catch((err) => { if (!res.headersSent) { - res.writeHead(500, JSON_HEADERS).end( - JSON.stringify({ error: String(err) }), - ); + res.writeHead(500, JSON_HEADERS).end(JSON.stringify({ error: String(err) })); } }); return; @@ -472,7 +554,7 @@ function handler(req: IncomingMessage, res: ServerResponse): void { export function startServer(port = PORT): ReturnType { const server = createServer(handler); - process.on("SIGTERM", () => { + process.on('SIGTERM', () => { server.close(() => process.exit(0)); }); diff --git a/packages/knative-server/test/authbridge-manifests.test.ts b/packages/knative-server/test/authbridge-manifests.test.ts index 8783126..6d0bd74 100644 --- a/packages/knative-server/test/authbridge-manifests.test.ts +++ b/packages/knative-server/test/authbridge-manifests.test.ts @@ -29,7 +29,9 @@ describe('ibac-stub manifest', () => { const docs = readDocs(IBAC_STUB_PATH); it('defines exactly one Deployment named ibac-stub', () => { - const deployments = docs.filter((d) => d?.kind === 'Deployment' && d?.metadata?.name === 'ibac-stub'); + const deployments = docs.filter( + (d) => d?.kind === 'Deployment' && d?.metadata?.name === 'ibac-stub', + ); expect(deployments).toHaveLength(1); }); @@ -38,8 +40,10 @@ describe('ibac-stub manifest', () => { expect(services).toHaveLength(1); }); - const deployment = docs.find((d) => d?.kind === 'Deployment' && d?.metadata?.name === 'ibac-stub') ?? {}; - const service = docs.find((d) => d?.kind === 'Service' && d?.metadata?.name === 'ibac-stub') ?? {}; + const deployment = + docs.find((d) => d?.kind === 'Deployment' && d?.metadata?.name === 'ibac-stub') ?? {}; + const service = + docs.find((d) => d?.kind === 'Service' && d?.metadata?.name === 'ibac-stub') ?? {}; const container = deployment.spec?.template?.spec?.containers?.[0] ?? {}; it("the Deployment's container exposes containerPort 8080", () => { @@ -66,26 +70,35 @@ describe('AB1 manifest', () => { const docs = readDocs(AB1_PATH); it('defines exactly one Deployment named authbridge-ab1', () => { - const deployments = docs.filter((d) => d?.kind === 'Deployment' && d?.metadata?.name === 'authbridge-ab1'); + const deployments = docs.filter( + (d) => d?.kind === 'Deployment' && d?.metadata?.name === 'authbridge-ab1', + ); expect(deployments).toHaveLength(1); }); it('defines exactly one Service named authbridge-ab1 with a port 8080', () => { - const services = docs.filter((d) => d?.kind === 'Service' && d?.metadata?.name === 'authbridge-ab1'); + const services = docs.filter( + (d) => d?.kind === 'Service' && d?.metadata?.name === 'authbridge-ab1', + ); expect(services).toHaveLength(1); expect(services[0].spec?.ports).toMatchObject([{ port: 8080 }]); }); it('defines exactly one ConfigMap named authbridge-ab1-config', () => { - const configMaps = docs.filter((d) => d?.kind === 'ConfigMap' && d?.metadata?.name === 'authbridge-ab1-config'); + const configMaps = docs.filter( + (d) => d?.kind === 'ConfigMap' && d?.metadata?.name === 'authbridge-ab1-config', + ); expect(configMaps).toHaveLength(1); }); - const deployment = docs.find((d) => d?.kind === 'Deployment' && d?.metadata?.name === 'authbridge-ab1') ?? {}; - const configMap = docs.find((d) => d?.kind === 'ConfigMap' && d?.metadata?.name === 'authbridge-ab1-config') ?? {}; + const deployment = + docs.find((d) => d?.kind === 'Deployment' && d?.metadata?.name === 'authbridge-ab1') ?? {}; + const configMap = + docs.find((d) => d?.kind === 'ConfigMap' && d?.metadata?.name === 'authbridge-ab1-config') ?? + {}; const container = deployment.spec?.template?.spec?.containers?.[0] ?? {}; - it("the Deployment container image is the official kext authbridge image", () => { + it('the Deployment container image is the official kext authbridge image', () => { expect(container.image).toBe('ghcr.io/rossoctl/kagenti-extensions/authbridge:main-9c131ee'); }); @@ -102,13 +115,17 @@ describe('AB1 manifest', () => { }); it('static-inject keys off the static Anthropic backend, not the inbound Host', () => { - const staticInject = config.pipeline.inbound.plugins.find((p: any) => p.name === 'static-inject'); + const staticInject = config.pipeline.inbound.plugins.find( + (p: any) => p.name === 'static-inject', + ); expect(staticInject.config.key_by).toBe('static'); expect(staticInject.config.key).toBe('api.anthropic.com'); }); it('static-inject injects the credential into the x-api-key header', () => { - const staticInject = config.pipeline.inbound.plugins.find((p: any) => p.name === 'static-inject'); + const staticInject = config.pipeline.inbound.plugins.find( + (p: any) => p.name === 'static-inject', + ); expect(staticInject.config.inject_header).toBe('x-api-key'); }); }); @@ -119,17 +136,23 @@ describe('echo-target manifest', () => { const docs = readDocs(ECHO_TARGET_PATH); it('defines exactly one Deployment named echo-target', () => { - const deployments = docs.filter((d) => d?.kind === 'Deployment' && d?.metadata?.name === 'echo-target'); + const deployments = docs.filter( + (d) => d?.kind === 'Deployment' && d?.metadata?.name === 'echo-target', + ); expect(deployments).toHaveLength(1); }); it('defines exactly one Service named echo-target', () => { - const services = docs.filter((d) => d?.kind === 'Service' && d?.metadata?.name === 'echo-target'); + const services = docs.filter( + (d) => d?.kind === 'Service' && d?.metadata?.name === 'echo-target', + ); expect(services).toHaveLength(1); }); - const deployment = docs.find((d) => d?.kind === 'Deployment' && d?.metadata?.name === 'echo-target') ?? {}; - const service = docs.find((d) => d?.kind === 'Service' && d?.metadata?.name === 'echo-target') ?? {}; + const deployment = + docs.find((d) => d?.kind === 'Deployment' && d?.metadata?.name === 'echo-target') ?? {}; + const service = + docs.find((d) => d?.kind === 'Service' && d?.metadata?.name === 'echo-target') ?? {}; it('the Service exposes port 80 targeting containerPort 8080', () => { expect(service.spec?.ports).toMatchObject([{ port: 80, targetPort: 8080 }]); @@ -146,11 +169,15 @@ describe('AB2 manifest', () => { const docs = readDocs(AB2_PATH); it('defines exactly one ConfigMap named authbridge-ab2-config', () => { - const configMaps = docs.filter((d) => d?.kind === 'ConfigMap' && d?.metadata?.name === 'authbridge-ab2-config'); + const configMaps = docs.filter( + (d) => d?.kind === 'ConfigMap' && d?.metadata?.name === 'authbridge-ab2-config', + ); expect(configMaps).toHaveLength(1); }); - const configMap = docs.find((d) => d?.kind === 'ConfigMap' && d?.metadata?.name === 'authbridge-ab2-config') ?? {}; + const configMap = + docs.find((d) => d?.kind === 'ConfigMap' && d?.metadata?.name === 'authbridge-ab2-config') ?? + {}; describe('embedded AB2 config (ConfigMap data["config.yaml"])', () => { const config = parse(configMap.data?.['config.yaml'] ?? ''); @@ -175,7 +202,9 @@ describe('AB2 manifest', () => { }); it('static-inject keys off the destination host, with no inject_header override', () => { - const staticInject = config.pipeline.outbound.plugins.find((p: any) => p.name === 'static-inject'); + const staticInject = config.pipeline.outbound.plugins.find( + (p: any) => p.name === 'static-inject', + ); expect(staticInject.config.key_by).toBe('host'); expect(staticInject.config).not.toHaveProperty('inject_header'); }); @@ -197,61 +226,85 @@ describe('sandbox-pool-ab2 manifest (SH_AUTHBRIDGE variant)', () => { expect(sandboxes.map((s) => s.metadata?.name)).toEqual(['sandbox-0', 'sandbox-1', 'sandbox-2']); }); - it.each(sandboxes.map((s, i) => [i, s]))('sandbox %s: pod has an authbridge-ab2 sidecar container', (_i, sandbox: any) => { - const containers = sandbox.spec?.podTemplate?.spec?.containers ?? []; - const ab2 = findContainer(containers, 'authbridge-ab2'); - expect(ab2.image).toBe('ghcr.io/rossoctl/kagenti-extensions/authbridge:main-9c131ee'); - expect(ab2.args).toEqual(['--config', '/etc/authbridge/config.yaml']); - }); - - it.each(sandboxes.map((s, i) => [i, s]))('sandbox %s: sandbox container is first and sets HTTP(S)_PROXY', (_i, sandbox: any) => { - const containers = sandbox.spec?.podTemplate?.spec?.containers ?? []; - expect(containers[0]?.name).toBe('sandbox'); - const envNames = Object.fromEntries((containers[0]?.env ?? []).map((e: any) => [e.name, e.value])); - expect(envNames.HTTP_PROXY).toBe('http://localhost:8081'); - expect(envNames.HTTPS_PROXY).toBe('http://localhost:8081'); - // curl ignores uppercase HTTP_PROXY for http:// URLs (only lowercase http_proxy is honored - // there); both cases must be set so the sandbox's plain `curl http://...` egress actually - // transits AB2. - expect(envNames.http_proxy).toBe('http://localhost:8081'); - expect(envNames.https_proxy).toBe('http://localhost:8081'); - }); - - it.each(sandboxes.map((s, i) => [i, s]))('sandbox %s: sandbox container uses the pre-baked image (no apk-at-startup)', (_i, sandbox: any) => { - const containers = sandbox.spec?.podTemplate?.spec?.containers ?? []; - const sandboxContainer = findContainer(containers, 'sandbox'); - // Tools (bash, coreutils, findutils, grep, ripgrep, git, curl) are baked into the image - // (deploy/knative/sandbox.Dockerfile) at build time instead of `apk add`-ed at container - // startup — apk-at-startup was slow/racy under load (2-3 min) and had to bypass the AB2 - // proxy env below to reach the Alpine CDN. No such bypass is needed anymore. - expect(sandboxContainer.image).toBe('dev.local/sandbox-rc1:rc1'); - expect(sandboxContainer.imagePullPolicy).toBe('IfNotPresent'); - const command = (sandboxContainer.command ?? []).join(' '); - expect(command).toContain('sleep infinity'); - expect(command).not.toContain('apk'); - }); - - it.each(sandboxes.map((s, i) => [i, s]))('sandbox %s: pod volumes mount the AB2 config and creds', (_i, sandbox: any) => { - const volumes = sandbox.spec?.podTemplate?.spec?.volumes ?? []; - const config = volumes.find((v: any) => v.name === 'config'); - const creds = volumes.find((v: any) => v.name === 'creds'); - expect(config?.configMap?.name).toBe('authbridge-ab2-config'); - expect(creds?.secret?.secretName).toBe('ab2-egress-cred'); - }); - - it.each(sandboxes.map((s, i) => [i, s]))('sandbox %s: sandbox container does NOT mount the creds secret (secret-free invariant)', (_i, sandbox: any) => { - const containers = sandbox.spec?.podTemplate?.spec?.containers ?? []; - const sandboxContainer = findContainer(containers, 'sandbox'); - const mounts = sandboxContainer.volumeMounts ?? []; - expect(mounts.some((m: any) => m.name === 'creds' || m.mountPath === '/etc/authbridge/creds')).toBe(false); - }); - - it.each(sandboxes.map((s, i) => [i, s]))('sandbox %s: authbridge-ab2 sidecar DOES mount the creds secret', (_i, sandbox: any) => { - const containers = sandbox.spec?.podTemplate?.spec?.containers ?? []; - const ab2 = findContainer(containers, 'authbridge-ab2'); - const mounts = ab2.volumeMounts ?? []; - expect(mounts.some((m: any) => m.name === 'creds' && m.mountPath === '/etc/authbridge/creds')).toBe(true); - }); + it.each(sandboxes.map((s, i) => [i, s]))( + 'sandbox %s: pod has an authbridge-ab2 sidecar container', + (_i, sandbox: any) => { + const containers = sandbox.spec?.podTemplate?.spec?.containers ?? []; + const ab2 = findContainer(containers, 'authbridge-ab2'); + expect(ab2.image).toBe('ghcr.io/rossoctl/kagenti-extensions/authbridge:main-9c131ee'); + expect(ab2.args).toEqual(['--config', '/etc/authbridge/config.yaml']); + }, + ); + + it.each(sandboxes.map((s, i) => [i, s]))( + 'sandbox %s: sandbox container is first and sets HTTP(S)_PROXY', + (_i, sandbox: any) => { + const containers = sandbox.spec?.podTemplate?.spec?.containers ?? []; + expect(containers[0]?.name).toBe('sandbox'); + const envNames = Object.fromEntries( + (containers[0]?.env ?? []).map((e: any) => [e.name, e.value]), + ); + expect(envNames.HTTP_PROXY).toBe('http://localhost:8081'); + expect(envNames.HTTPS_PROXY).toBe('http://localhost:8081'); + // curl ignores uppercase HTTP_PROXY for http:// URLs (only lowercase http_proxy is honored + // there); both cases must be set so the sandbox's plain `curl http://...` egress actually + // transits AB2. + expect(envNames.http_proxy).toBe('http://localhost:8081'); + expect(envNames.https_proxy).toBe('http://localhost:8081'); + }, + ); + + it.each(sandboxes.map((s, i) => [i, s]))( + 'sandbox %s: sandbox container uses the pre-baked image (no apk-at-startup)', + (_i, sandbox: any) => { + const containers = sandbox.spec?.podTemplate?.spec?.containers ?? []; + const sandboxContainer = findContainer(containers, 'sandbox'); + // Tools (bash, coreutils, findutils, grep, ripgrep, git, curl) are baked into the image + // (deploy/knative/sandbox.Dockerfile) at build time instead of `apk add`-ed at container + // startup — apk-at-startup was slow/racy under load (2-3 min) and had to bypass the AB2 + // proxy env below to reach the Alpine CDN. No such bypass is needed anymore. + expect(sandboxContainer.image).toBe('dev.local/sandbox-rc1:rc1'); + expect(sandboxContainer.imagePullPolicy).toBe('IfNotPresent'); + const command = (sandboxContainer.command ?? []).join(' '); + expect(command).toContain('sleep infinity'); + expect(command).not.toContain('apk'); + }, + ); + + it.each(sandboxes.map((s, i) => [i, s]))( + 'sandbox %s: pod volumes mount the AB2 config and creds', + (_i, sandbox: any) => { + const volumes = sandbox.spec?.podTemplate?.spec?.volumes ?? []; + const config = volumes.find((v: any) => v.name === 'config'); + const creds = volumes.find((v: any) => v.name === 'creds'); + expect(config?.configMap?.name).toBe('authbridge-ab2-config'); + expect(creds?.secret?.secretName).toBe('ab2-egress-cred'); + }, + ); + + it.each(sandboxes.map((s, i) => [i, s]))( + 'sandbox %s: sandbox container does NOT mount the creds secret (secret-free invariant)', + (_i, sandbox: any) => { + const containers = sandbox.spec?.podTemplate?.spec?.containers ?? []; + const sandboxContainer = findContainer(containers, 'sandbox'); + const mounts = sandboxContainer.volumeMounts ?? []; + expect( + mounts.some((m: any) => m.name === 'creds' || m.mountPath === '/etc/authbridge/creds'), + ).toBe(false); + }, + ); + + it.each(sandboxes.map((s, i) => [i, s]))( + 'sandbox %s: authbridge-ab2 sidecar DOES mount the creds secret', + (_i, sandbox: any) => { + const containers = sandbox.spec?.podTemplate?.spec?.containers ?? []; + const ab2 = findContainer(containers, 'authbridge-ab2'); + const mounts = ab2.volumeMounts ?? []; + expect( + mounts.some((m: any) => m.name === 'creds' && m.mountPath === '/etc/authbridge/creds'), + ).toBe(true); + }, + ); }); // Cross-file invariant (issue #103): setup-ocp.sh's SH_AUTHBRIDGE=1 pool pre-poll computes its @@ -265,7 +318,9 @@ describe('sandbox-pool-ab2 manifest (SH_AUTHBRIDGE variant)', () => { describe('setup-ocp.sh AB2 pool pre-poll sandbox count (issue #103)', () => { const script = readFileSync(resolve(DEPLOY, 'setup-ocp.sh'), 'utf8'); const lines = script.split('\n'); - const sandboxes = readDocs(resolve(DEPLOY, 'sandbox-pool-ab2.yaml')).filter((d) => d?.kind === 'Sandbox'); + const sandboxes = readDocs(resolve(DEPLOY, 'sandbox-pool-ab2.yaml')).filter( + (d) => d?.kind === 'Sandbox', + ); /** Resolve a shell selector token to its literal value, following one level of `VAR=...`. */ function resolveSelectorToken(token: string): string { diff --git a/packages/knative-server/test/context-service.test.ts b/packages/knative-server/test/context-service.test.ts index ad88502..5bd3952 100644 --- a/packages/knative-server/test/context-service.test.ts +++ b/packages/knative-server/test/context-service.test.ts @@ -1,6 +1,6 @@ -import { afterEach, describe, expect, it, vi } from "vitest"; +import { afterEach, describe, expect, it, vi } from 'vitest'; -import { contextServiceConfigured, createWorkload } from "../src/context-service.js"; +import { contextServiceConfigured, createWorkload } from '../src/context-service.js'; afterEach(() => { vi.useRealTimers(); @@ -8,68 +8,92 @@ afterEach(() => { vi.unstubAllGlobals(); }); -describe("Context Service client", () => { - it("is disabled unless CONTEXT_SERVICE_URL is explicitly set", () => { - vi.stubEnv("CONTEXT_SERVICE_URL", ""); +describe('Context Service client', () => { + it('is disabled unless CONTEXT_SERVICE_URL is explicitly set', () => { + vi.stubEnv('CONTEXT_SERVICE_URL', ''); expect(contextServiceConfigured()).toBe(false); }); - it("creates managed shared storage through the configured service", async () => { - vi.stubEnv("CONTEXT_SERVICE_URL", "http://context.example/"); - const fetch = vi.fn().mockResolvedValue(new Response(JSON.stringify({ - name: "demo", status: "provisioning", replicas: 3, readyReplicas: 0, - sandboxSelector: "context.rossoctl.io/pool=demo", - workspace: { size: "5Gi", accessMode: "ReadWriteMany", storageClass: "ibm-scale-csi" }, - }), { status: 201 })); - vi.stubGlobal("fetch", fetch); + it('creates managed shared storage through the configured service', async () => { + vi.stubEnv('CONTEXT_SERVICE_URL', 'http://context.example/'); + const fetch = vi.fn().mockResolvedValue( + new Response( + JSON.stringify({ + name: 'demo', + status: 'provisioning', + replicas: 3, + readyReplicas: 0, + sandboxSelector: 'context.rossoctl.io/pool=demo', + workspace: { size: '5Gi', accessMode: 'ReadWriteMany', storageClass: 'ibm-scale-csi' }, + }), + { status: 201 }, + ), + ); + vi.stubGlobal('fetch', fetch); - await createWorkload("demo", { + await createWorkload('demo', { sandboxes: 3, - workspace: { shared: true, size: "5Gi", storageClass: "ibm-scale-csi" }, + workspace: { shared: true, size: '5Gi', storageClass: 'ibm-scale-csi' }, }); - expect(fetch.mock.calls[0][0]).toBe("http://context.example/v1/sandbox-pools"); + expect(fetch.mock.calls[0][0]).toBe('http://context.example/v1/sandbox-pools'); const init = fetch.mock.calls[0][1] as RequestInit; expect(init.signal).toBeInstanceOf(AbortSignal); expect(JSON.parse(String(init.body))).toEqual({ - name: "demo", + name: 'demo', replicas: 3, - workspace: { size: "5Gi", accessMode: "ReadWriteMany", storageClass: "ibm-scale-csi" }, + workspace: { size: '5Gi', accessMode: 'ReadWriteMany', storageClass: 'ibm-scale-csi' }, }); }); - it("passes an existing read-only claim without managed-workspace fields", async () => { - vi.stubEnv("CONTEXT_SERVICE_URL", "http://context.example"); - const fetch = vi.fn().mockResolvedValue(new Response(JSON.stringify({ - name: "readers", status: "provisioning", replicas: 4, readyReplicas: 0, - sandboxSelector: "context.rossoctl.io/pool=readers", - workspace: { claimName: "mosaic", readOnly: true }, - }), { status: 201 })); - vi.stubGlobal("fetch", fetch); + it('passes an existing read-only claim without managed-workspace fields', async () => { + vi.stubEnv('CONTEXT_SERVICE_URL', 'http://context.example'); + const fetch = vi.fn().mockResolvedValue( + new Response( + JSON.stringify({ + name: 'readers', + status: 'provisioning', + replicas: 4, + readyReplicas: 0, + sandboxSelector: 'context.rossoctl.io/pool=readers', + workspace: { claimName: 'mosaic', readOnly: true }, + }), + { status: 201 }, + ), + ); + vi.stubGlobal('fetch', fetch); - await createWorkload("readers", { + await createWorkload('readers', { sandboxes: 4, - workspace: { claimName: "mosaic", readOnly: true }, + workspace: { claimName: 'mosaic', readOnly: true }, }); const init = fetch.mock.calls[0][1] as RequestInit; expect(JSON.parse(String(init.body))).toEqual({ - name: "readers", + name: 'readers', replicas: 4, - workspace: { claimName: "mosaic", readOnly: true }, + workspace: { claimName: 'mosaic', readOnly: true }, }); }); - it("aborts a Context Service request after the configured timeout", async () => { + it('aborts a Context Service request after the configured timeout', async () => { vi.useFakeTimers(); - vi.stubEnv("CONTEXT_SERVICE_URL", "http://context.example"); - vi.stubEnv("CONTEXT_SERVICE_TIMEOUT_MS", "25"); - vi.stubGlobal("fetch", vi.fn((_url: string, init?: RequestInit) => new Promise((_resolve, reject) => { - init?.signal?.addEventListener("abort", () => reject(new DOMException("Aborted", "AbortError"))); - }))); + vi.stubEnv('CONTEXT_SERVICE_URL', 'http://context.example'); + vi.stubEnv('CONTEXT_SERVICE_TIMEOUT_MS', '25'); + vi.stubGlobal( + 'fetch', + vi.fn( + (_url: string, init?: RequestInit) => + new Promise((_resolve, reject) => { + init?.signal?.addEventListener('abort', () => + reject(new DOMException('Aborted', 'AbortError')), + ); + }), + ), + ); - const request = createWorkload("demo", {}); - const rejected = expect(request).rejects.toMatchObject({ name: "AbortError" }); + const request = createWorkload('demo', {}); + const rejected = expect(request).rejects.toMatchObject({ name: 'AbortError' }); await vi.advanceTimersByTimeAsync(25); await rejected; }); diff --git a/packages/knative-server/test/cron-dispatch.test.ts b/packages/knative-server/test/cron-dispatch.test.ts index 8bf5e6e..ff17f35 100644 --- a/packages/knative-server/test/cron-dispatch.test.ts +++ b/packages/knative-server/test/cron-dispatch.test.ts @@ -1,90 +1,105 @@ -import { describe, it, expect, vi, beforeEach, afterEach } from "vitest"; -import { mkdtempSync, writeFileSync, rmSync } from "node:fs"; -import { join } from "node:path"; -import { tmpdir } from "node:os"; -import { applyFire, dispatchAll, loadConfig, exitCodeFor } from "../src/cron-dispatch"; +import { describe, it, expect, vi, beforeEach, afterEach } from 'vitest'; +import { mkdtempSync, writeFileSync, rmSync } from 'node:fs'; +import { join } from 'node:path'; +import { tmpdir } from 'node:os'; +import { applyFire, dispatchAll, loadConfig, exitCodeFor } from '../src/cron-dispatch'; -describe("applyFire", () => { - it("substitutes every __FIRE__ occurrence in string fields and passes others through", () => { +describe('applyFire', () => { + it('substitutes every __FIRE__ occurrence in string fields and passes others through', () => { const out = applyFire( - { sessionId: "nightly/__FIRE__/i1", model: "claude-haiku-4-5", item: { item_id: "i1", file: "risky.py", pattern: "eval(" } }, - "fire-42", + { + sessionId: 'nightly/__FIRE__/i1', + model: 'claude-haiku-4-5', + item: { item_id: 'i1', file: 'risky.py', pattern: 'eval(' }, + }, + 'fire-42', ); expect(out).toEqual({ - sessionId: "nightly/fire-42/i1", - model: "claude-haiku-4-5", - item: { item_id: "i1", file: "risky.py", pattern: "eval(" }, + sessionId: 'nightly/fire-42/i1', + model: 'claude-haiku-4-5', + item: { item_id: 'i1', file: 'risky.py', pattern: 'eval(' }, }); }); - it("does not mutate the input object", () => { - const input = { sessionId: "s/__FIRE__" }; - applyFire(input, "f1"); - expect(input.sessionId).toBe("s/__FIRE__"); + it('does not mutate the input object', () => { + const input = { sessionId: 's/__FIRE__' }; + applyFire(input, 'f1'); + expect(input.sessionId).toBe('s/__FIRE__'); }); }); -describe("dispatchAll", () => { +describe('dispatchAll', () => { const ITEMS = [ - { sessionId: "n/__FIRE__/i1", item: { item_id: "i1", file: "risky.py", pattern: "eval(" } }, - { sessionId: "n/__FIRE__/i2", item: { item_id: "i2", file: "safe.py", pattern: "eval(" } }, + { sessionId: 'n/__FIRE__/i1', item: { item_id: 'i1', file: 'risky.py', pattern: 'eval(' } }, + { sessionId: 'n/__FIRE__/i2', item: { item_id: 'i2', file: 'safe.py', pattern: 'eval(' } }, ]; - it("posts every item once with async:true and __FIRE__ substituted; all accepted", async () => { + it('posts every item once with async:true and __FIRE__ substituted; all accepted', async () => { const seen: Record[] = []; - const post = vi.fn(async (env: Record) => { seen.push(env); return true; }); - const r = await dispatchAll(ITEMS, "fire-1", post); + const post = vi.fn(async (env: Record) => { + seen.push(env); + return true; + }); + const r = await dispatchAll(ITEMS, 'fire-1', post); expect(r).toEqual({ total: 2, accepted: 2, failed: 0 }); expect(post).toHaveBeenCalledTimes(2); - expect(seen[0]).toMatchObject({ sessionId: "n/fire-1/i1", item: { item_id: "i1", file: "risky.py" }, async: true }); - expect(seen[1]).toMatchObject({ sessionId: "n/fire-1/i2", async: true }); + expect(seen[0]).toMatchObject({ + sessionId: 'n/fire-1/i1', + item: { item_id: 'i1', file: 'risky.py' }, + async: true, + }); + expect(seen[1]).toMatchObject({ sessionId: 'n/fire-1/i2', async: true }); }); - it("counts a rejected post as failed but still attempts the rest", async () => { - const post = vi.fn() - .mockResolvedValueOnce(false) - .mockResolvedValueOnce(true); - const r = await dispatchAll(ITEMS, "fire-1", post as any); + it('counts a rejected post as failed but still attempts the rest', async () => { + const post = vi.fn().mockResolvedValueOnce(false).mockResolvedValueOnce(true); + const r = await dispatchAll(ITEMS, 'fire-1', post as any); expect(r).toEqual({ total: 2, accepted: 1, failed: 1 }); expect(post).toHaveBeenCalledTimes(2); }); - it("counts a thrown post as failed and continues", async () => { - const post = vi.fn() - .mockRejectedValueOnce(new Error("network")) - .mockResolvedValueOnce(true); - const r = await dispatchAll(ITEMS, "fire-1", post as any); + it('counts a thrown post as failed and continues', async () => { + const post = vi.fn().mockRejectedValueOnce(new Error('network')).mockResolvedValueOnce(true); + const r = await dispatchAll(ITEMS, 'fire-1', post as any); expect(r).toEqual({ total: 2, accepted: 1, failed: 1 }); expect(post).toHaveBeenCalledTimes(2); }); - it("posts nothing for an empty list", async () => { + it('posts nothing for an empty list', async () => { const post = vi.fn(async () => true); - const r = await dispatchAll([], "fire-1", post); + const r = await dispatchAll([], 'fire-1', post); expect(r).toEqual({ total: 0, accepted: 0, failed: 0 }); expect(post).not.toHaveBeenCalled(); }); }); -describe("exitCodeFor", () => { - it("returns 0 when nothing failed", () => { expect(exitCodeFor({ failed: 0 })).toBe(0); }); - it("returns 1 when any item failed", () => { expect(exitCodeFor({ failed: 1 })).toBe(1); }); +describe('exitCodeFor', () => { + it('returns 0 when nothing failed', () => { + expect(exitCodeFor({ failed: 0 })).toBe(0); + }); + it('returns 1 when any item failed', () => { + expect(exitCodeFor({ failed: 1 })).toBe(1); + }); }); -describe("loadConfig", () => { +describe('loadConfig', () => { let dir: string; - beforeEach(() => { dir = mkdtempSync(join(tmpdir(), "cron-")); }); - afterEach(() => { rmSync(dir, { recursive: true, force: true }); }); - it("reads the items array", () => { - const p = join(dir, "schedule.json"); - writeFileSync(p, JSON.stringify({ items: [{ sessionId: "a" }, { sessionId: "b" }] })); - expect(loadConfig(p)).toEqual([{ sessionId: "a" }, { sessionId: "b" }]); + beforeEach(() => { + dir = mkdtempSync(join(tmpdir(), 'cron-')); + }); + afterEach(() => { + rmSync(dir, { recursive: true, force: true }); + }); + it('reads the items array', () => { + const p = join(dir, 'schedule.json'); + writeFileSync(p, JSON.stringify({ items: [{ sessionId: 'a' }, { sessionId: 'b' }] })); + expect(loadConfig(p)).toEqual([{ sessionId: 'a' }, { sessionId: 'b' }]); }); - it("throws when items is missing or not an array", () => { - const p = join(dir, "bad.json"); + it('throws when items is missing or not an array', () => { + const p = join(dir, 'bad.json'); writeFileSync(p, JSON.stringify({ nope: true })); expect(() => loadConfig(p)).toThrow(); }); - it("throws when the file is missing", () => { - expect(() => loadConfig(join(dir, "absent.json"))).toThrow(); + it('throws when the file is missing', () => { + expect(() => loadConfig(join(dir, 'absent.json'))).toThrow(); }); }); diff --git a/packages/knative-server/test/harness-egress-policy.test.ts b/packages/knative-server/test/harness-egress-policy.test.ts index 71308a7..85e9ed2 100644 --- a/packages/knative-server/test/harness-egress-policy.test.ts +++ b/packages/knative-server/test/harness-egress-policy.test.ts @@ -72,7 +72,9 @@ describe('harness egress NetworkPolicy manifest', () => { ), ); expect(dnsRule, 'a rule scoped to the openshift-dns namespace').toBeDefined(); - const protos = new Set((dnsRule.ports ?? []).filter((p: any) => p.port === 5353).map((p: any) => p.protocol)); + const protos = new Set( + (dnsRule.ports ?? []).filter((p: any) => p.port === 5353).map((p: any) => p.protocol), + ); expect(protos.has('UDP')).toBe(true); expect(protos.has('TCP')).toBe(true); }); @@ -90,7 +92,9 @@ describe('harness egress NetworkPolicy manifest', () => { ), ); expect(dnsRule, 'a rule scoped to the kube-system namespace').toBeDefined(); - const protos = new Set((dnsRule.ports ?? []).filter((p: any) => p.port === 53).map((p: any) => p.protocol)); + const protos = new Set( + (dnsRule.ports ?? []).filter((p: any) => p.port === 53).map((p: any) => p.protocol), + ); expect(protos.has('UDP')).toBe(true); expect(protos.has('TCP')).toBe(true); }); @@ -183,7 +187,9 @@ describe('tightened AB1 egress variant', () => { ), ); expect(dnsRule, 'a rule scoped to the openshift-dns namespace').toBeDefined(); - const protos = new Set((dnsRule.ports ?? []).filter((p: any) => p.port === 5353).map((p: any) => p.protocol)); + const protos = new Set( + (dnsRule.ports ?? []).filter((p: any) => p.port === 5353).map((p: any) => p.protocol), + ); expect(protos.has('UDP')).toBe(true); expect(protos.has('TCP')).toBe(true); }); @@ -201,7 +207,9 @@ describe('tightened AB1 egress variant', () => { ), ); expect(dnsRule, 'a rule scoped to the kube-system namespace').toBeDefined(); - const protos = new Set((dnsRule.ports ?? []).filter((p: any) => p.port === 53).map((p: any) => p.protocol)); + const protos = new Set( + (dnsRule.ports ?? []).filter((p: any) => p.port === 53).map((p: any) => p.protocol), + ); expect(protos.has('UDP')).toBe(true); expect(protos.has('TCP')).toBe(true); }); diff --git a/packages/knative-server/test/prompt-envelope.test.ts b/packages/knative-server/test/prompt-envelope.test.ts index d37169c..a672381 100644 --- a/packages/knative-server/test/prompt-envelope.test.ts +++ b/packages/knative-server/test/prompt-envelope.test.ts @@ -1,24 +1,28 @@ -import { describe, it, expect } from "vitest"; -import { isPromptEnvelope, isRunEnvelope } from "../src/server.js"; +import { describe, it, expect } from 'vitest'; +import { isPromptEnvelope, isRunEnvelope } from '../src/server.js'; -describe("isPromptEnvelope", () => { - const ok = { sessionId: "s", kind: "prompt", prompt: "summarize the repo" }; - it("accepts a well-formed prompt envelope", () => { expect(isPromptEnvelope(ok)).toBe(true); }); - it("rejects a missing/empty prompt", () => { - expect(isPromptEnvelope({ sessionId: "s", kind: "prompt" })).toBe(false); - expect(isPromptEnvelope({ sessionId: "s", kind: "prompt", prompt: 123 })).toBe(false); +describe('isPromptEnvelope', () => { + const ok = { sessionId: 's', kind: 'prompt', prompt: 'summarize the repo' }; + it('accepts a well-formed prompt envelope', () => { + expect(isPromptEnvelope(ok)).toBe(true); }); - it("rejects the wrong kind", () => { - expect(isPromptEnvelope({ sessionId: "s", kind: "solve", prompt: "x" })).toBe(false); + it('rejects a missing/empty prompt', () => { + expect(isPromptEnvelope({ sessionId: 's', kind: 'prompt' })).toBe(false); + expect(isPromptEnvelope({ sessionId: 's', kind: 'prompt', prompt: 123 })).toBe(false); }); - it("rejects a missing sessionId", () => { - expect(isPromptEnvelope({ kind: "prompt", prompt: "x" })).toBe(false); + it('rejects the wrong kind', () => { + expect(isPromptEnvelope({ sessionId: 's', kind: 'solve', prompt: 'x' })).toBe(false); + }); + it('rejects a missing sessionId', () => { + expect(isPromptEnvelope({ kind: 'prompt', prompt: 'x' })).toBe(false); }); }); -describe("isRunEnvelope accepts a prompt envelope", () => { - it("accepts a well-formed prompt envelope", () => { - expect(isRunEnvelope({ sessionId: "s", kind: "prompt", prompt: "x" })).toBe(true); +describe('isRunEnvelope accepts a prompt envelope', () => { + it('accepts a well-formed prompt envelope', () => { + expect(isRunEnvelope({ sessionId: 's', kind: 'prompt', prompt: 'x' })).toBe(true); + }); + it('still rejects junk', () => { + expect(isRunEnvelope({ foo: 1 })).toBe(false); }); - it("still rejects junk", () => { expect(isRunEnvelope({ foo: 1 })).toBe(false); }); }); diff --git a/packages/knative-server/test/relay-deployment.test.ts b/packages/knative-server/test/relay-deployment.test.ts index dbbe760..d023e4d 100644 --- a/packages/knative-server/test/relay-deployment.test.ts +++ b/packages/knative-server/test/relay-deployment.test.ts @@ -1,56 +1,62 @@ -import { readFileSync } from "node:fs"; -import { fileURLToPath } from "node:url"; -import { dirname, resolve } from "node:path"; -import { describe, expect, it } from "vitest"; -import { parse, parseAllDocuments } from "yaml"; +import { readFileSync } from 'node:fs'; +import { fileURLToPath } from 'node:url'; +import { dirname, resolve } from 'node:path'; +import { describe, expect, it } from 'vitest'; +import { parse, parseAllDocuments } from 'yaml'; type EnvVar = { name: string; value?: string }; -const REPO_ROOT = resolve(dirname(fileURLToPath(import.meta.url)), "../../.."); -const DEPLOY = resolve(REPO_ROOT, "deploy/knative"); -const docs = () => parseAllDocuments(readFileSync(resolve(DEPLOY, "relay-deployment.yaml"), "utf8")).map((d) => d.toJS()); +const REPO_ROOT = resolve(dirname(fileURLToPath(import.meta.url)), '../../..'); +const DEPLOY = resolve(REPO_ROOT, 'deploy/knative'); +const docs = () => + parseAllDocuments(readFileSync(resolve(DEPLOY, 'relay-deployment.yaml'), 'utf8')).map((d) => + d.toJS(), + ); -describe("relay-deployment.yaml", () => { - it("is a single-replica Deployment plus a Service", () => { +describe('relay-deployment.yaml', () => { + it('is a single-replica Deployment plus a Service', () => { const all = docs(); - const dep = all.find((o) => o.kind === "Deployment"); - const svc = all.find((o) => o.kind === "Service"); + const dep = all.find((o) => o.kind === 'Deployment'); + const svc = all.find((o) => o.kind === 'Service'); expect(dep?.spec.replicas).toBe(1); - expect(dep?.metadata.name).toBe("sandbox-relay"); - expect(dep?.spec.template.metadata.labels.app).toBe("sandbox-relay"); - expect(svc?.spec.selector.app).toBe("sandbox-relay"); + expect(dep?.metadata.name).toBe('sandbox-relay'); + expect(dep?.spec.template.metadata.labels.app).toBe('sandbox-relay'); + expect(svc?.spec.selector.app).toBe('sandbox-relay'); }); - it("runs the relay entrypoint from its package dir so tsx + deps resolve (issue #102 follow-up)", () => { - const dep = docs().find((o) => o.kind === "Deployment"); + it('runs the relay entrypoint from its package dir so tsx + deps resolve (issue #102 follow-up)', () => { + const dep = docs().find((o) => o.kind === 'Deployment'); const c = dep.spec.template.spec.containers[0]; - expect(c.image).toContain("serverless-harness"); + expect(c.image).toContain('serverless-harness'); // `node --import tsx` resolves the tsx loader relative to the CWD, and the published // image links tsx only into packages/sandbox-relay/node_modules (no /app/node_modules // hoist). A CWD of /app crashes ERR_MODULE_NOT_FOUND 'tsx'; run from the package dir. - expect(c.workingDir).toBe("/app/packages/sandbox-relay"); - const cmd = c.command.join(" "); - expect(cmd).toContain("--import tsx"); - expect(cmd).toContain("src/main.ts"); + expect(c.workingDir).toBe('/app/packages/sandbox-relay'); + const cmd = c.command.join(' '); + expect(cmd).toContain('--import tsx'); + expect(cmd).toContain('src/main.ts'); }); - it("is referenced by both kustomizations", () => { - const base = parse(readFileSync(resolve(DEPLOY, "kustomization.yaml"), "utf8")); - const ocp = parse(readFileSync(resolve(DEPLOY, "overlays/ocp/kustomization.yaml"), "utf8")); - expect(base.resources).toContain("relay-deployment.yaml"); - expect(ocp.resources).toContain("../../relay-deployment.yaml"); + it('is referenced by both kustomizations', () => { + const base = parse(readFileSync(resolve(DEPLOY, 'kustomization.yaml'), 'utf8')); + const ocp = parse(readFileSync(resolve(DEPLOY, 'overlays/ocp/kustomization.yaml'), 'utf8')); + expect(base.resources).toContain('relay-deployment.yaml'); + expect(ocp.resources).toContain('../../relay-deployment.yaml'); }); - it("sets SH_RELAY_TOKEN matching the worker example (relay auth is fail-closed)", () => { - const c = docs().find((o) => o.kind === "Deployment").spec.template.spec.containers[0]; + it('sets SH_RELAY_TOKEN matching the worker example (relay auth is fail-closed)', () => { + const c = docs().find((o) => o.kind === 'Deployment').spec.template.spec.containers[0]; const env: EnvVar[] = c.env; - const token = env.find((e) => e.name === "SH_RELAY_TOKEN"); - expect(token?.value, "relay auth is fail-closed (main.ts's makeDefaultValidateToken): with no SH_RELAY_TOKEN set, every worker Attach is rejected").toBeTruthy(); - const worker = parse(readFileSync(resolve(DEPLOY, "worker-example.yaml"), "utf8")); + const token = env.find((e) => e.name === 'SH_RELAY_TOKEN'); + expect( + token?.value, + "relay auth is fail-closed (main.ts's makeDefaultValidateToken): with no SH_RELAY_TOKEN set, every worker Attach is rejected", + ).toBeTruthy(); + const worker = parse(readFileSync(resolve(DEPLOY, 'worker-example.yaml'), 'utf8')); const wEnv: EnvVar[] = worker.spec.template.spec.containers[0].env; expect( token!.value, "SH_RELAY_TOKEN must equal worker-example.yaml's SANDBOX_TOKEN, or the relay rejects every Attach from a worker deployed off that example", - ).toBe(wEnv.find((e) => e.name === "SANDBOX_TOKEN")!.value); + ).toBe(wEnv.find((e) => e.name === 'SANDBOX_TOKEN')!.value); }); }); diff --git a/packages/knative-server/test/run-leaf-async-route.test.ts b/packages/knative-server/test/run-leaf-async-route.test.ts index 7e8e956..52b5398 100644 --- a/packages/knative-server/test/run-leaf-async-route.test.ts +++ b/packages/knative-server/test/run-leaf-async-route.test.ts @@ -1,93 +1,135 @@ // packages/knative-server/test/run-leaf-async-route.test.ts -import { describe, it, expect, vi, beforeEach } from "vitest"; +import { describe, it, expect, vi, beforeEach } from 'vitest'; // Keep the result store hermetic — no live Redis in unit tests. -vi.mock("@sh/harness/leaf-result-store", async (orig) => { - const actual = await orig(); +vi.mock('@sh/harness/leaf-result-store', async (orig) => { + const actual = await orig(); const mem = new Map(); - class FakeStore { async set(k: string, v: string) { mem.set(k, v); } async get(k: string) { return mem.get(k) ?? null; } async close() {} } + class FakeStore { + async set(k: string, v: string) { + mem.set(k, v); + } + async get(k: string) { + return mem.get(k) ?? null; + } + async close() {} + } return { ...actual, RedisResultStore: FakeStore }; }); -const enqueue = vi.fn(async () => "1-0"); +const enqueue = vi.fn(async () => '1-0'); const ensureGroup = vi.fn(async () => {}); -vi.mock("@sh/work-queue", () => ({ - RedisWorkQueue: class { ensureGroup = ensureGroup; enqueue = enqueue; close = async () => {}; }, +vi.mock('@sh/work-queue', () => ({ + RedisWorkQueue: class { + ensureGroup = ensureGroup; + enqueue = enqueue; + close = async () => {}; + }, })); -vi.mock("@sh/harness/run-leaf", () => ({ +vi.mock('@sh/harness/run-leaf', () => ({ runLeaf: vi.fn(), - validateItem: (o: any) => (o && typeof o.item_id === "string" && typeof o.file === "string" && typeof o.pattern === "string" ? o : null), - leafSessionId: (env: any) => (env.sessionId ?? "leaf").replace(/[^A-Za-z0-9._-]/g, "-").replace(/^[^A-Za-z0-9]+|[^A-Za-z0-9]+$/g, "") || "leaf", + validateItem: (o: any) => + o && + typeof o.item_id === 'string' && + typeof o.file === 'string' && + typeof o.pattern === 'string' + ? o + : null, + leafSessionId: (env: any) => + (env.sessionId ?? 'leaf') + .replace(/[^A-Za-z0-9._-]/g, '-') + .replace(/^[^A-Za-z0-9]+|[^A-Za-z0-9]+$/g, '') || 'leaf', })); -import { startServer } from "../src/server"; +import { startServer } from '../src/server'; -let base: string; let server: any; -beforeEach(() => { enqueue.mockClear(); }); +let base: string; +let server: any; +beforeEach(() => { + enqueue.mockClear(); +}); async function req(method: string, path: string, body?: unknown) { - const res = await fetch(base + path, { method, headers: { "content-type": "application/json" }, body: body ? JSON.stringify(body) : undefined }); + const res = await fetch(base + path, { + method, + headers: { 'content-type': 'application/json' }, + body: body ? JSON.stringify(body) : undefined, + }); return { status: res.status, json: await res.json().catch(() => ({})) }; } -describe("async /runs", () => { - beforeEach(() => { server = startServer(0); base = `http://127.0.0.1:${server.address().port}`; }); +describe('async /runs', () => { + beforeEach(() => { + server = startServer(0); + base = `http://127.0.0.1:${server.address().port}`; + }); - it("202 + enqueues on async:true with a valid envelope", async () => { - const r = await req("POST", "/runs", { sessionId: "run/i1", item: { item_id: "i1", file: "f", pattern: "p" }, async: true }); + it('202 + enqueues on async:true with a valid envelope', async () => { + const r = await req('POST', '/runs', { + sessionId: 'run/i1', + item: { item_id: 'i1', file: 'f', pattern: 'p' }, + async: true, + }); expect(r.status).toBe(202); - expect(r.json).toMatchObject({ status: "accepted", sessionId: "run/i1" }); - expect(r.json).not.toHaveProperty("doneMarker"); - expect(r.json).not.toHaveProperty("resultRef"); + expect(r.json).toMatchObject({ status: 'accepted', sessionId: 'run/i1' }); + expect(r.json).not.toHaveProperty('doneMarker'); + expect(r.json).not.toHaveProperty('resultRef'); expect(enqueue).toHaveBeenCalledOnce(); server.close(); }); - it("400 and no enqueue on a malformed async envelope", async () => { - const r = await req("POST", "/runs", { sessionId: "s", async: true }); + it('400 and no enqueue on a malformed async envelope', async () => { + const r = await req('POST', '/runs', { sessionId: 's', async: true }); expect(r.status).toBe(400); expect(enqueue).not.toHaveBeenCalled(); server.close(); }); - it("status returns queued when no record exists", async () => { - const r = await req("GET", "/runs/status?sessionId=run/none"); + it('status returns queued when no record exists', async () => { + const r = await req('GET', '/runs/status?sessionId=run/none'); expect(r.status).toBe(200); - expect(r.json).toMatchObject({ status: "queued" }); + expect(r.json).toMatchObject({ status: 'queued' }); server.close(); }); - it("status returns 400 when sessionId is missing", async () => { - const r = await req("GET", "/runs/status"); + it('status returns 400 when sessionId is missing', async () => { + const r = await req('GET', '/runs/status'); expect(r.status).toBe(400); - expect(r.json).toMatchObject({ error: "sessionId_required" }); + expect(r.json).toMatchObject({ error: 'sessionId_required' }); server.close(); }); }); // Regression: the pre-rename async paths must keep working as deprecated aliases (issue #37). -describe("deprecated /run-leaf aliases", () => { - beforeEach(() => { server = startServer(0); base = `http://127.0.0.1:${server.address().port}`; }); +describe('deprecated /run-leaf aliases', () => { + beforeEach(() => { + server = startServer(0); + base = `http://127.0.0.1:${server.address().port}`; + }); - it("POST /run-leaf still enqueues async work and warns about deprecation", async () => { - const warn = vi.spyOn(console, "warn").mockImplementation(() => {}); - const r = await req("POST", "/run-leaf", { sessionId: "run/i1", item: { item_id: "i1", file: "f", pattern: "p" }, async: true }); + it('POST /run-leaf still enqueues async work and warns about deprecation', async () => { + const warn = vi.spyOn(console, 'warn').mockImplementation(() => {}); + const r = await req('POST', '/run-leaf', { + sessionId: 'run/i1', + item: { item_id: 'i1', file: 'f', pattern: 'p' }, + async: true, + }); expect(r.status).toBe(202); - expect(r.json).toMatchObject({ status: "accepted", sessionId: "run/i1" }); + expect(r.json).toMatchObject({ status: 'accepted', sessionId: 'run/i1' }); expect(enqueue).toHaveBeenCalledOnce(); expect(warn).toHaveBeenCalled(); - expect(warn.mock.calls.map((c) => String(c[0])).join("\n")).toContain("/run-leaf"); + expect(warn.mock.calls.map((c) => String(c[0])).join('\n')).toContain('/run-leaf'); warn.mockRestore(); server.close(); }); - it("GET /run-leaf/status warns about deprecation and returns queued for unknown session", async () => { - const warn = vi.spyOn(console, "warn").mockImplementation(() => {}); - const r = await req("GET", "/run-leaf/status?sessionId=run/i1"); + it('GET /run-leaf/status warns about deprecation and returns queued for unknown session', async () => { + const warn = vi.spyOn(console, 'warn').mockImplementation(() => {}); + const r = await req('GET', '/run-leaf/status?sessionId=run/i1'); expect(r.status).toBe(200); - expect(r.json).toMatchObject({ status: "queued" }); + expect(r.json).toMatchObject({ status: 'queued' }); expect(warn).toHaveBeenCalled(); - expect(warn.mock.calls.map((c) => String(c[0])).join("\n")).toContain("/run-leaf/status"); + expect(warn.mock.calls.map((c) => String(c[0])).join('\n')).toContain('/run-leaf/status'); warn.mockRestore(); server.close(); }); diff --git a/packages/knative-server/test/run-leaf-route.test.ts b/packages/knative-server/test/run-leaf-route.test.ts index 28817ee..f98da70 100644 --- a/packages/knative-server/test/run-leaf-route.test.ts +++ b/packages/knative-server/test/run-leaf-route.test.ts @@ -1,57 +1,98 @@ -import { describe, it, expect, vi, beforeEach, afterEach } from "vitest"; +import { describe, it, expect, vi, beforeEach, afterEach } from 'vitest'; // Keep the result store hermetic — no live Redis in unit tests. `resultSet` records every // write so tests can assert whether a result record was persisted (it must NOT be on 503). const resultSet = vi.fn(); -vi.mock("@sh/harness/leaf-result-store", async (orig) => { - const actual = await orig(); +vi.mock('@sh/harness/leaf-result-store', async (orig) => { + const actual = await orig(); const mem = new Map(); - class FakeStore { async set(k: string, v: string) { resultSet(k, v); mem.set(k, v); } async get(k: string) { return mem.get(k) ?? null; } async close() {} } + class FakeStore { + async set(k: string, v: string) { + resultSet(k, v); + mem.set(k, v); + } + async get(k: string) { + return mem.get(k) ?? null; + } + async close() {} + } return { ...actual, RedisResultStore: FakeStore }; }); const runLeaf = vi.fn(); -vi.mock("@sh/harness/run-leaf", () => ({ +vi.mock('@sh/harness/run-leaf', () => ({ runLeaf: (...a: any[]) => runLeaf(...a), - validateItem: (o: any) => (o && typeof o.item_id === "string" && typeof o.file === "string" && typeof o.pattern === "string" ? o : null), - leafSessionId: (env: any) => (env.sessionId ?? "leaf").replace(/[^A-Za-z0-9._-]/g, "-").replace(/^[^A-Za-z0-9]+|[^A-Za-z0-9]+$/g, "") || "leaf", + validateItem: (o: any) => + o && + typeof o.item_id === 'string' && + typeof o.file === 'string' && + typeof o.pattern === 'string' + ? o + : null, + leafSessionId: (env: any) => + (env.sessionId ?? 'leaf') + .replace(/[^A-Za-z0-9._-]/g, '-') + .replace(/^[^A-Za-z0-9]+|[^A-Za-z0-9]+$/g, '') || 'leaf', })); -import { startServer } from "../src/server"; +import { startServer } from '../src/server'; -let base: string; let server: any; -beforeEach(() => { runLeaf.mockReset(); resultSet.mockReset(); }); +let base: string; +let server: any; +beforeEach(() => { + runLeaf.mockReset(); + resultSet.mockReset(); +}); async function post(path: string, body: unknown) { const res = await fetch(base + path, { - method: "POST", headers: { "content-type": "application/json" }, body: JSON.stringify(body), + method: 'POST', + headers: { 'content-type': 'application/json' }, + body: JSON.stringify(body), }); return { status: res.status, headers: res.headers, json: await res.json().catch(() => ({})) }; } -describe("POST /runs", () => { - beforeEach(() => { server = startServer(0); base = `http://127.0.0.1:${server.address().port}`; }); - afterEach(() => { server.close(); }); +describe('POST /runs', () => { + beforeEach(() => { + server = startServer(0); + base = `http://127.0.0.1:${server.address().port}`; + }); + afterEach(() => { + server.close(); + }); - it("400s on a malformed envelope (missing item)", async () => { - const r = await post("/runs", { sessionId: "s" }); + it('400s on a malformed envelope (missing item)', async () => { + const r = await post('/runs', { sessionId: 's' }); expect(r.status).toBe(400); }); - it("returns the verdict inline on a valid envelope", async () => { - runLeaf.mockResolvedValue({ status: "done", verdict: { item_id: "i1", verdict: "CLEAR", reason: "ok" } }); - const r = await post("/runs", { sessionId: "run/i1", item: { item_id: "i1", file: "f", pattern: "p" } }); + it('returns the verdict inline on a valid envelope', async () => { + runLeaf.mockResolvedValue({ + status: 'done', + verdict: { item_id: 'i1', verdict: 'CLEAR', reason: 'ok' }, + }); + const r = await post('/runs', { + sessionId: 'run/i1', + item: { item_id: 'i1', file: 'f', pattern: 'p' }, + }); expect(r.status).toBe(200); - expect(r.json).toEqual({ status: "done", verdict: { item_id: "i1", verdict: "CLEAR", reason: "ok" } }); + expect(r.json).toEqual({ + status: 'done', + verdict: { item_id: 'i1', verdict: 'CLEAR', reason: 'ok' }, + }); expect(runLeaf).toHaveBeenCalledOnce(); }); - it("does not accept a sandbox selector from the run request", async () => { - runLeaf.mockResolvedValue({ status: "done", verdict: { item_id: "i1", verdict: "CLEAR", reason: "ok" } }); - await post("/runs", { - sessionId: "run/i1", - sandboxPoolSelector: "attacker.example/pool=other", - item: { item_id: "i1", file: "f", pattern: "p" }, + it('does not accept a sandbox selector from the run request', async () => { + runLeaf.mockResolvedValue({ + status: 'done', + verdict: { item_id: 'i1', verdict: 'CLEAR', reason: 'ok' }, + }); + await post('/runs', { + sessionId: 'run/i1', + sandboxPoolSelector: 'attacker.example/pool=other', + item: { item_id: 'i1', file: 'f', pattern: 'p' }, }); expect(runLeaf).toHaveBeenCalledWith( @@ -62,14 +103,15 @@ describe("POST /runs", () => { }); // Spec §4.3: the sync path must bound-wait with backoff on pool saturation, then 503 Retry-After. -describe("POST /runs saturation (spec §4.3)", () => { +describe('POST /runs saturation (spec §4.3)', () => { beforeEach(() => { // Tiny budget keeps the test fast; the loop re-attempts pool acquisition every few ms. - process.env.KAGENTI_SYNC_SATURATION_WAIT_MS = "60"; - process.env.KAGENTI_SYNC_SATURATION_BACKOFF_MS = "5"; - process.env.KAGENTI_SYNC_SATURATION_MAX_BACKOFF_MS = "10"; - process.env.KAGENTI_SYNC_SATURATION_RETRY_AFTER_S = "7"; - server = startServer(0); base = `http://127.0.0.1:${server.address().port}`; + process.env.KAGENTI_SYNC_SATURATION_WAIT_MS = '60'; + process.env.KAGENTI_SYNC_SATURATION_BACKOFF_MS = '5'; + process.env.KAGENTI_SYNC_SATURATION_MAX_BACKOFF_MS = '10'; + process.env.KAGENTI_SYNC_SATURATION_RETRY_AFTER_S = '7'; + server = startServer(0); + base = `http://127.0.0.1:${server.address().port}`; }); afterEach(() => { server.close(); @@ -79,24 +121,34 @@ describe("POST /runs saturation (spec §4.3)", () => { delete process.env.KAGENTI_SYNC_SATURATION_RETRY_AFTER_S; }); - it("returns 503 with Retry-After after the wait budget is exhausted", async () => { - runLeaf.mockResolvedValue({ status: "failed", reason: "saturated" }); - const r = await post("/runs", { sessionId: "run/i1", item: { item_id: "i1", file: "f", pattern: "p" } }); + it('returns 503 with Retry-After after the wait budget is exhausted', async () => { + runLeaf.mockResolvedValue({ status: 'failed', reason: 'saturated' }); + const r = await post('/runs', { + sessionId: 'run/i1', + item: { item_id: 'i1', file: 'f', pattern: 'p' }, + }); expect(r.status).toBe(503); - expect(r.headers.get("retry-after")).toBe("7"); + expect(r.headers.get('retry-after')).toBe('7'); // Bounded WAIT with BACKOFF means the pool was re-attempted at least once before giving up. expect(runLeaf.mock.calls.length).toBeGreaterThan(1); // A 503 means "retry", not a terminal failure — no result record must be persisted. expect(resultSet).not.toHaveBeenCalled(); }); - it("retries pool acquisition and returns the verdict once a pod frees", async () => { - runLeaf - .mockResolvedValueOnce({ status: "failed", reason: "saturated" }) - .mockResolvedValueOnce({ status: "done", verdict: { item_id: "i1", verdict: "CLEAR", reason: "ok" } }); - const r = await post("/runs", { sessionId: "run/i1", item: { item_id: "i1", file: "f", pattern: "p" } }); + it('retries pool acquisition and returns the verdict once a pod frees', async () => { + runLeaf.mockResolvedValueOnce({ status: 'failed', reason: 'saturated' }).mockResolvedValueOnce({ + status: 'done', + verdict: { item_id: 'i1', verdict: 'CLEAR', reason: 'ok' }, + }); + const r = await post('/runs', { + sessionId: 'run/i1', + item: { item_id: 'i1', file: 'f', pattern: 'p' }, + }); expect(r.status).toBe(200); - expect(r.json).toEqual({ status: "done", verdict: { item_id: "i1", verdict: "CLEAR", reason: "ok" } }); + expect(r.json).toEqual({ + status: 'done', + verdict: { item_id: 'i1', verdict: 'CLEAR', reason: 'ok' }, + }); expect(runLeaf).toHaveBeenCalledTimes(2); expect(resultSet).toHaveBeenCalledOnce(); }); @@ -104,7 +156,7 @@ describe("POST /runs saturation (spec §4.3)", () => { // A malformed operator knob must not silently disable the bounded wait or emit "Retry-After: NaN". // Each *_MS/*_S value falls back to its default when it is not a finite, non-negative number. -describe("POST /runs saturation env hardening", () => { +describe('POST /runs saturation env hardening', () => { afterEach(() => { server.close(); delete process.env.KAGENTI_SYNC_SATURATION_WAIT_MS; @@ -113,51 +165,74 @@ describe("POST /runs saturation env hardening", () => { delete process.env.KAGENTI_SYNC_SATURATION_RETRY_AFTER_S; }); - it("falls back to the default budget on a malformed wait value (still bound-waits)", async () => { + it('falls back to the default budget on a malformed wait value (still bound-waits)', async () => { // Bug being guarded: "abc" → NaN → `Date.now() < NaN` is false → the loop is skipped and a // saturated first result becomes an immediate 503. With the fallback, WAIT_MS reverts to its // (ample) default, so a pod that frees on the second attempt is served a 200. - process.env.KAGENTI_SYNC_SATURATION_WAIT_MS = "abc"; - process.env.KAGENTI_SYNC_SATURATION_BACKOFF_MS = "5"; - process.env.KAGENTI_SYNC_SATURATION_MAX_BACKOFF_MS = "10"; - server = startServer(0); base = `http://127.0.0.1:${server.address().port}`; - runLeaf - .mockResolvedValueOnce({ status: "failed", reason: "saturated" }) - .mockResolvedValueOnce({ status: "done", verdict: { item_id: "i1", verdict: "CLEAR", reason: "ok" } }); - const r = await post("/runs", { sessionId: "run/i1", item: { item_id: "i1", file: "f", pattern: "p" } }); + process.env.KAGENTI_SYNC_SATURATION_WAIT_MS = 'abc'; + process.env.KAGENTI_SYNC_SATURATION_BACKOFF_MS = '5'; + process.env.KAGENTI_SYNC_SATURATION_MAX_BACKOFF_MS = '10'; + server = startServer(0); + base = `http://127.0.0.1:${server.address().port}`; + runLeaf.mockResolvedValueOnce({ status: 'failed', reason: 'saturated' }).mockResolvedValueOnce({ + status: 'done', + verdict: { item_id: 'i1', verdict: 'CLEAR', reason: 'ok' }, + }); + const r = await post('/runs', { + sessionId: 'run/i1', + item: { item_id: 'i1', file: 'f', pattern: 'p' }, + }); expect(r.status).toBe(200); expect(runLeaf).toHaveBeenCalledTimes(2); }); - it("advertises the default Retry-After when the env value is malformed (never NaN)", async () => { - process.env.KAGENTI_SYNC_SATURATION_WAIT_MS = "30"; - process.env.KAGENTI_SYNC_SATURATION_BACKOFF_MS = "5"; - process.env.KAGENTI_SYNC_SATURATION_MAX_BACKOFF_MS = "10"; - process.env.KAGENTI_SYNC_SATURATION_RETRY_AFTER_S = "abc"; - server = startServer(0); base = `http://127.0.0.1:${server.address().port}`; - runLeaf.mockResolvedValue({ status: "failed", reason: "saturated" }); - const r = await post("/runs", { sessionId: "run/i1", item: { item_id: "i1", file: "f", pattern: "p" } }); + it('advertises the default Retry-After when the env value is malformed (never NaN)', async () => { + process.env.KAGENTI_SYNC_SATURATION_WAIT_MS = '30'; + process.env.KAGENTI_SYNC_SATURATION_BACKOFF_MS = '5'; + process.env.KAGENTI_SYNC_SATURATION_MAX_BACKOFF_MS = '10'; + process.env.KAGENTI_SYNC_SATURATION_RETRY_AFTER_S = 'abc'; + server = startServer(0); + base = `http://127.0.0.1:${server.address().port}`; + runLeaf.mockResolvedValue({ status: 'failed', reason: 'saturated' }); + const r = await post('/runs', { + sessionId: 'run/i1', + item: { item_id: 'i1', file: 'f', pattern: 'p' }, + }); expect(r.status).toBe(503); - expect(r.headers.get("retry-after")).toBe("5"); + expect(r.headers.get('retry-after')).toBe('5'); }); }); // Regression: the pre-rename path must keep working as a deprecated alias (issue #37). -describe("POST /run-leaf (deprecated alias)", () => { - beforeEach(() => { server = startServer(0); base = `http://127.0.0.1:${server.address().port}`; }); - afterEach(() => { server.close(); }); - - it("still dispatches to runLeaf and warns about deprecation", async () => { - const warn = vi.spyOn(console, "warn").mockImplementation(() => {}); - runLeaf.mockResolvedValue({ status: "done", verdict: { item_id: "i1", verdict: "CLEAR", reason: "ok" } }); - const r = await post("/run-leaf", { sessionId: "run/i1", item: { item_id: "i1", file: "f", pattern: "p" } }); +describe('POST /run-leaf (deprecated alias)', () => { + beforeEach(() => { + server = startServer(0); + base = `http://127.0.0.1:${server.address().port}`; + }); + afterEach(() => { + server.close(); + }); + + it('still dispatches to runLeaf and warns about deprecation', async () => { + const warn = vi.spyOn(console, 'warn').mockImplementation(() => {}); + runLeaf.mockResolvedValue({ + status: 'done', + verdict: { item_id: 'i1', verdict: 'CLEAR', reason: 'ok' }, + }); + const r = await post('/run-leaf', { + sessionId: 'run/i1', + item: { item_id: 'i1', file: 'f', pattern: 'p' }, + }); expect(r.status).toBe(200); - expect(r.json).toEqual({ status: "done", verdict: { item_id: "i1", verdict: "CLEAR", reason: "ok" } }); + expect(r.json).toEqual({ + status: 'done', + verdict: { item_id: 'i1', verdict: 'CLEAR', reason: 'ok' }, + }); expect(runLeaf).toHaveBeenCalledOnce(); expect(warn).toHaveBeenCalled(); - const msg = warn.mock.calls.map((c) => String(c[0])).join("\n"); - expect(msg).toContain("/run-leaf"); - expect(msg).toContain("/runs"); + const msg = warn.mock.calls.map((c) => String(c[0])).join('\n'); + expect(msg).toContain('/run-leaf'); + expect(msg).toContain('/runs'); warn.mockRestore(); }); }); diff --git a/packages/knative-server/test/server.test.ts b/packages/knative-server/test/server.test.ts index 3bab19e..9b7d8ff 100644 --- a/packages/knative-server/test/server.test.ts +++ b/packages/knative-server/test/server.test.ts @@ -1,29 +1,46 @@ -import { describe, it, expect, beforeAll, beforeEach, afterAll, vi } from "vitest"; -import http from "node:http"; +import { describe, it, expect, beforeAll, beforeEach, afterAll, vi } from 'vitest'; +import http from 'node:http'; // Mock runTurn before importing server -vi.mock("@sh/harness/run-turn", () => ({ +vi.mock('@sh/harness/run-turn', () => ({ runTurn: vi.fn(), executeTurn: vi.fn(), })); // Keep the result store hermetic — no live Redis in unit tests. -vi.mock("@sh/harness/leaf-result-store", async (orig) => { - const actual = await orig(); +vi.mock('@sh/harness/leaf-result-store', async (orig) => { + const actual = await orig(); const mem = new Map(); - class FakeStore { async set(k: string, v: string) { mem.set(k, v); } async get(k: string) { return mem.get(k) ?? null; } async close() {} } + class FakeStore { + async set(k: string, v: string) { + mem.set(k, v); + } + async get(k: string) { + return mem.get(k) ?? null; + } + async close() {} + } return { ...actual, RedisResultStore: FakeStore }; }); const runLeaf = vi.fn(); -vi.mock("@sh/harness/run-leaf", () => ({ +vi.mock('@sh/harness/run-leaf', () => ({ runLeaf: (...a: any[]) => runLeaf(...a), - validateItem: (o: any) => (o && typeof o.item_id === "string" && typeof o.file === "string" && typeof o.pattern === "string" ? o : null), - leafSessionId: (env: any) => (env.sessionId ?? "leaf").replace(/[^A-Za-z0-9._-]/g, "-").replace(/^[^A-Za-z0-9]+|[^A-Za-z0-9]+$/g, "") || "leaf", + validateItem: (o: any) => + o && + typeof o.item_id === 'string' && + typeof o.file === 'string' && + typeof o.pattern === 'string' + ? o + : null, + leafSessionId: (env: any) => + (env.sessionId ?? 'leaf') + .replace(/[^A-Za-z0-9._-]/g, '-') + .replace(/^[^A-Za-z0-9]+|[^A-Za-z0-9]+$/g, '') || 'leaf', })); -import { startServer } from "../src/server.js"; -import { runTurn, executeTurn } from "@sh/harness/run-turn"; +import { startServer } from '../src/server.js'; +import { runTurn, executeTurn } from '@sh/harness/run-turn'; const mockedRunTurn = vi.mocked(runTurn); const mockedExecuteTurn = vi.mocked(executeTurn); @@ -39,17 +56,17 @@ function request( const url = new URL(path, baseUrl); const req = http.request(url, { method }, (res) => { const chunks: Buffer[] = []; - res.on("data", (c: Buffer) => chunks.push(c)); - res.on("end", () => + res.on('data', (c: Buffer) => chunks.push(c)); + res.on('end', () => resolve({ status: res.statusCode ?? 0, body: Buffer.concat(chunks).toString(), }), ); }); - req.on("error", reject); + req.on('error', reject); if (body !== undefined) { - req.setHeader("Content-Type", "application/json"); + req.setHeader('Content-Type', 'application/json'); req.write(JSON.stringify(body)); } req.end(); @@ -62,23 +79,23 @@ function sseRequest( body: unknown, ): Promise<{ status: number; contentType: string | undefined; raw: string }> { return new Promise((resolve, reject) => { - const url = new URL("/turn", baseUrl); + const url = new URL('/turn', baseUrl); const req = http.request( url, - { method: "POST", headers: { "Content-Type": "application/json", ...headers } }, + { method: 'POST', headers: { 'Content-Type': 'application/json', ...headers } }, (res) => { const chunks: Buffer[] = []; - res.on("data", (c: Buffer) => chunks.push(c)); - res.on("end", () => + res.on('data', (c: Buffer) => chunks.push(c)); + res.on('end', () => resolve({ status: res.statusCode ?? 0, - contentType: res.headers["content-type"], + contentType: res.headers['content-type'], raw: Buffer.concat(chunks).toString(), }), ); }, ); - req.on("error", reject); + req.on('error', reject); req.write(JSON.stringify(body)); req.end(); }); @@ -86,16 +103,18 @@ function sseRequest( async function post(path: string, body: unknown): Promise<{ status: number; json: any }> { const res = await fetch(baseUrl + path, { - method: "POST", headers: { "content-type": "application/json" }, body: JSON.stringify(body), + method: 'POST', + headers: { 'content-type': 'application/json' }, + body: JSON.stringify(body), }); return { status: res.status, json: await res.json().catch(() => ({})) }; } beforeAll(async () => { server = startServer(0); // port 0 = random available port - await new Promise((resolve) => server.on("listening", resolve)); + await new Promise((resolve) => server.on('listening', resolve)); const addr = server.address(); - if (addr && typeof addr === "object") { + if (addr && typeof addr === 'object') { baseUrl = `http://127.0.0.1:${addr.port}`; } }); @@ -104,115 +123,112 @@ afterAll(async () => { await new Promise((resolve) => server.close(() => resolve())); }); -describe("GET /health", () => { - it("returns 200 ok", async () => { - const res = await request("GET", "/health"); +describe('GET /health', () => { + it('returns 200 ok', async () => { + const res = await request('GET', '/health'); expect(res.status).toBe(200); - expect(res.body).toBe("ok"); + expect(res.body).toBe('ok'); }); }); -describe("POST /turn", () => { - it("returns 200 with session result on success", async () => { +describe('POST /turn', () => { + it('returns 200 with session result on success', async () => { mockedRunTurn.mockResolvedValueOnce({ - sessionId: "test-session-1", - response: "Hello!", - stopReason: "end_turn", + sessionId: 'test-session-1', + response: 'Hello!', + stopReason: 'end_turn', }); - const res = await request("POST", "/turn", { prompt: "Hi" }); + const res = await request('POST', '/turn', { prompt: 'Hi' }); expect(res.status).toBe(200); const json = JSON.parse(res.body); - expect(json.sessionId).toBe("test-session-1"); - expect(json.response).toBe("Hello!"); - expect(json.stopReason).toBe("end_turn"); + expect(json.sessionId).toBe('test-session-1'); + expect(json.response).toBe('Hello!'); + expect(json.stopReason).toBe('end_turn'); }); - it("passes sessionId to runTurn when provided", async () => { + it('passes sessionId to runTurn when provided', async () => { mockedRunTurn.mockResolvedValueOnce({ - sessionId: "existing-session", - response: "Resumed!", - stopReason: "end_turn", + sessionId: 'existing-session', + response: 'Resumed!', + stopReason: 'end_turn', }); - const res = await request("POST", "/turn", { - sessionId: "existing-session", - prompt: "Continue", + const res = await request('POST', '/turn', { + sessionId: 'existing-session', + prompt: 'Continue', }); expect(res.status).toBe(200); - expect(mockedRunTurn).toHaveBeenCalledWith( - "Continue", - "existing-session", - expect.any(Object), - ); + expect(mockedRunTurn).toHaveBeenCalledWith('Continue', 'existing-session', expect.any(Object)); }); - it("returns 400 when prompt is missing", async () => { - const res = await request("POST", "/turn", { sessionId: "abc" }); + it('returns 400 when prompt is missing', async () => { + const res = await request('POST', '/turn', { sessionId: 'abc' }); expect(res.status).toBe(400); - expect(JSON.parse(res.body).error).toBe("prompt_required"); + expect(JSON.parse(res.body).error).toBe('prompt_required'); }); - it("returns 400 on invalid JSON", async () => { - const res = await new Promise<{ status: number; body: string }>( - (resolve, reject) => { - const url = new URL("/turn", baseUrl); - const req = http.request(url, { method: "POST" }, (r) => { - const chunks: Buffer[] = []; - r.on("data", (c: Buffer) => chunks.push(c)); - r.on("end", () => - resolve({ - status: r.statusCode ?? 0, - body: Buffer.concat(chunks).toString(), - }), - ); - }); - req.on("error", reject); - req.write("not valid json{{{"); - req.end(); - }, - ); + it('returns 400 on invalid JSON', async () => { + const res = await new Promise<{ status: number; body: string }>((resolve, reject) => { + const url = new URL('/turn', baseUrl); + const req = http.request(url, { method: 'POST' }, (r) => { + const chunks: Buffer[] = []; + r.on('data', (c: Buffer) => chunks.push(c)); + r.on('end', () => + resolve({ + status: r.statusCode ?? 0, + body: Buffer.concat(chunks).toString(), + }), + ); + }); + req.on('error', reject); + req.write('not valid json{{{'); + req.end(); + }); expect(res.status).toBe(400); - expect(JSON.parse(res.body).error).toBe("invalid_json"); + expect(JSON.parse(res.body).error).toBe('invalid_json'); }); - it("returns 404 when session not found", async () => { + it('returns 404 when session not found', async () => { mockedRunTurn.mockRejectedValueOnce( - new Error("Cannot resume: no session in backend for id xyz"), + new Error('Cannot resume: no session in backend for id xyz'), ); - const res = await request("POST", "/turn", { - sessionId: "xyz", - prompt: "hello", + const res = await request('POST', '/turn', { + sessionId: 'xyz', + prompt: 'hello', }); expect(res.status).toBe(404); - expect(JSON.parse(res.body).error).toBe("session_not_found"); + expect(JSON.parse(res.body).error).toBe('session_not_found'); }); - it("returns 500 on unexpected errors", async () => { - mockedRunTurn.mockRejectedValueOnce(new Error("LLM timeout")); + it('returns 500 on unexpected errors', async () => { + mockedRunTurn.mockRejectedValueOnce(new Error('LLM timeout')); - const res = await request("POST", "/turn", { prompt: "hello" }); + const res = await request('POST', '/turn', { prompt: 'hello' }); expect(res.status).toBe(500); - expect(JSON.parse(res.body).error).toBe("LLM timeout"); + expect(JSON.parse(res.body).error).toBe('LLM timeout'); }); }); -describe("GET /runs/status", () => { - it("GET /runs/status returns queued when no record exists", async () => { +describe('GET /runs/status', () => { + it('GET /runs/status returns queued when no record exists', async () => { const r = await (await fetch(`${baseUrl}/runs/status?sessionId=run/none`)).json(); - expect(r).toEqual({ status: "queued" }); + expect(r).toEqual({ status: 'queued' }); }); - it("GET /runs/status returns the record after a sync run", async () => { - runLeaf.mockResolvedValue({ status: "done", verdict: { item_id: "i1", verdict: "FLAGGED", reason: "x" } }); - await post("/runs", { sessionId: "run/i1", item: { item_id: "i1", file: "f", pattern: "p" } }); + it('GET /runs/status returns the record after a sync run', async () => { + runLeaf.mockResolvedValue({ + status: 'done', + verdict: { item_id: 'i1', verdict: 'FLAGGED', reason: 'x' }, + }); + await post('/runs', { sessionId: 'run/i1', item: { item_id: 'i1', file: 'f', pattern: 'p' } }); const r = await (await fetch(`${baseUrl}/runs/status?sessionId=run/i1`)).json(); - expect(r).toMatchObject({ status: "done", verdict: { item_id: "i1", verdict: "FLAGGED" } }); + expect(r).toMatchObject({ status: 'done', verdict: { item_id: 'i1', verdict: 'FLAGGED' } }); }); }); -describe("POST /turn — back-compat & streaming", () => { +describe('POST /turn — back-compat & streaming', () => { // Isolate call-count assertions (not.toHaveBeenCalled) from prior tests: this package's vitest // config sets no clearMocks, so clear per test. Implementations are set inside each test after this. beforeEach(() => { @@ -220,10 +236,10 @@ describe("POST /turn — back-compat & streaming", () => { mockedRunTurn.mockClear(); }); - it("no Accept header → golden byte-for-byte sync JSON (the back-compat linchpin)", async () => { - const result = { sessionId: "gold-1", response: "Hi there", stopReason: "end_turn" }; + it('no Accept header → golden byte-for-byte sync JSON (the back-compat linchpin)', async () => { + const result = { sessionId: 'gold-1', response: 'Hi there', stopReason: 'end_turn' }; mockedRunTurn.mockResolvedValueOnce(result as any); - const res = await request("POST", "/turn", { prompt: "Hi" }); + const res = await request('POST', '/turn', { prompt: 'Hi' }); expect(res.status).toBe(200); // Frozen wire bytes (not JSON.stringify(result)): pins the server's own sync-response emission, // so a future edit to how the JSON path serializes/orders keys fails here (back-compat, ADR-0029). @@ -232,48 +248,51 @@ describe("POST /turn — back-compat & streaming", () => { expect(mockedExecuteTurn).not.toHaveBeenCalled(); // sync path never touches executeTurn }); - it("Accept: text/event-stream → SSE content-type, ordered frames, terminal done", async () => { + it('Accept: text/event-stream → SSE content-type, ordered frames, terminal done', async () => { mockedExecuteTurn.mockImplementationOnce(async (input: any) => { - input.onEvent?.({ type: "text", delta: "Hel" }); - input.onEvent?.({ type: "text", delta: "lo" }); - input.onEvent?.({ type: "tool_use", id: "t1", name: "bash", args: { cmd: "ls" } }); - input.onEvent?.({ type: "tool_result", id: "t1", isError: false, preview: "file.txt" }); - return { sessionId: "s-stream", response: "Hello", stopReason: "end_turn" }; + input.onEvent?.({ type: 'text', delta: 'Hel' }); + input.onEvent?.({ type: 'text', delta: 'lo' }); + input.onEvent?.({ type: 'tool_use', id: 't1', name: 'bash', args: { cmd: 'ls' } }); + input.onEvent?.({ type: 'tool_result', id: 't1', isError: false, preview: 'file.txt' }); + return { sessionId: 's-stream', response: 'Hello', stopReason: 'end_turn' }; }); - const res = await sseRequest({ Accept: "text/event-stream" }, { prompt: "Hi" }); + const res = await sseRequest({ Accept: 'text/event-stream' }, { prompt: 'Hi' }); expect(res.status).toBe(200); - expect(res.contentType).toBe("text/event-stream"); - const events = res.raw.split("\n\n").filter((b) => b.startsWith("event:")); + expect(res.contentType).toBe('text/event-stream'); + const events = res.raw.split('\n\n').filter((b) => b.startsWith('event:')); expect(events[0]).toBe('event: text\ndata: {"type":"text","delta":"Hel"}'); expect(res.raw).toContain( 'event: tool_use\ndata: {"type":"tool_use","id":"t1","name":"bash","args":{"cmd":"ls"}}', ); const last = events.at(-1)!; - expect(last.startsWith("event: done")).toBe(true); + expect(last.startsWith('event: done')).toBe(true); expect(last).toContain('"sessionId":"s-stream"'); expect(last).toContain('"stopReason":"end_turn"'); }); - it("bad sessionId + streaming Accept → real 404 JSON, not an error frame (pre-first-frame)", async () => { + it('bad sessionId + streaming Accept → real 404 JSON, not an error frame (pre-first-frame)', async () => { mockedExecuteTurn.mockRejectedValueOnce( - new Error("Cannot resume: no session in backend for id xyz"), + new Error('Cannot resume: no session in backend for id xyz'), + ); + const res = await sseRequest( + { Accept: 'text/event-stream' }, + { sessionId: 'xyz', prompt: 'hi' }, ); - const res = await sseRequest({ Accept: "text/event-stream" }, { sessionId: "xyz", prompt: "hi" }); expect(res.status).toBe(404); - expect(res.contentType).toBe("application/json"); + expect(res.contentType).toBe('application/json'); const parsed = JSON.parse(res.raw); - expect(parsed.error).toBe("session_not_found"); - expect(parsed.sessionId).toBe("xyz"); + expect(parsed.error).toBe('session_not_found'); + expect(parsed.sessionId).toBe('xyz'); }); - it("missing prompt + streaming Accept → 400 prompt_required (pre-flight, before the branch)", async () => { - const res = await sseRequest({ Accept: "text/event-stream" }, { sessionId: "abc" }); + it('missing prompt + streaming Accept → 400 prompt_required (pre-flight, before the branch)', async () => { + const res = await sseRequest({ Accept: 'text/event-stream' }, { sessionId: 'abc' }); expect(res.status).toBe(400); - expect(JSON.parse(res.raw).error).toBe("prompt_required"); + expect(JSON.parse(res.raw).error).toBe('prompt_required'); expect(mockedExecuteTurn).not.toHaveBeenCalled(); }); - it("client disconnect mid-stream aborts the executeTurn signal", async () => { + it('client disconnect mid-stream aborts the executeTurn signal', async () => { let capturedSignal: AbortSignal | undefined; let sawFirstFrame: (() => void) | undefined; const firstFrame = new Promise((r) => { @@ -281,22 +300,22 @@ describe("POST /turn — back-compat & streaming", () => { }); mockedExecuteTurn.mockImplementationOnce((input: any) => { capturedSignal = input.signal; - input.onEvent?.({ type: "text", delta: "partial" }); + input.onEvent?.({ type: 'text', delta: 'partial' }); sawFirstFrame?.(); // Resolve only once aborted, mimicking session.abort() unwinding the turn. return new Promise((resolve) => { - input.signal?.addEventListener("abort", () => - resolve({ sessionId: "s-abort", response: "partial", stopReason: "aborted" }), + input.signal?.addEventListener('abort', () => + resolve({ sessionId: 's-abort', response: 'partial', stopReason: 'aborted' }), ); }); }); - const url = new URL("/turn", baseUrl); + const url = new URL('/turn', baseUrl); const req = http.request(url, { - method: "POST", - headers: { "Content-Type": "application/json", Accept: "text/event-stream" }, + method: 'POST', + headers: { 'Content-Type': 'application/json', Accept: 'text/event-stream' }, }); - req.on("error", () => {}); // the deliberate req.destroy() below hangs up the socket mid-response - req.write(JSON.stringify({ sessionId: "s-abort", prompt: "hi" })); + req.on('error', () => {}); // the deliberate req.destroy() below hangs up the socket mid-response + req.write(JSON.stringify({ sessionId: 's-abort', prompt: 'hi' })); req.end(); await firstFrame; // server-side promise resolved by the mock after the first frame req.destroy(); // client disconnect @@ -305,34 +324,34 @@ describe("POST /turn — back-compat & streaming", () => { }); }); - it("executeTurn rejects AFTER a frame flushed → 200 + terminal error frame, not 500 JSON (regime 3)", async () => { + it('executeTurn rejects AFTER a frame flushed → 200 + terminal error frame, not 500 JSON (regime 3)', async () => { // The only net-new failure surface in ADR-0029: once ≥1 frame is on the wire the 200 status is // spent, so a mid-turn failure can no longer become a 500 JSON body — it must degrade to a // terminal `event: error` frame carrying the same facts (§3.4 regime 3). mockedExecuteTurn.mockImplementationOnce(async (input: any) => { - input.onEvent?.({ type: "text", delta: "partial" }); - throw new Error("LLM exploded mid-stream"); + input.onEvent?.({ type: 'text', delta: 'partial' }); + throw new Error('LLM exploded mid-stream'); }); - const res = await sseRequest({ Accept: "text/event-stream" }, { prompt: "hi" }); + const res = await sseRequest({ Accept: 'text/event-stream' }, { prompt: 'hi' }); expect(res.status).toBe(200); // headers committed by the first frame — never rewritten to 500 - expect(res.contentType).toBe("text/event-stream"); + expect(res.contentType).toBe('text/event-stream'); expect(res.raw).toContain('event: text\ndata: {"type":"text","delta":"partial"}'); - const events = res.raw.split("\n\n").filter((b) => b.startsWith("event:")); + const events = res.raw.split('\n\n').filter((b) => b.startsWith('event:')); const last = events.at(-1)!; - expect(last.startsWith("event: error")).toBe(true); - const data = JSON.parse(last.slice(last.indexOf("data: ") + "data: ".length)); + expect(last.startsWith('event: error')).toBe(true); + const data = JSON.parse(last.slice(last.indexOf('data: ') + 'data: '.length)); expect(data).toMatchObject({ - type: "error", - sessionId: "", // no sessionId on a fresh turn → "" on the wire (server.ts:193) - stopReason: "error", - errorMessage: "LLM exploded mid-stream", + type: 'error', + sessionId: '', // no sessionId on a fresh turn → "" on the wire (server.ts:193) + stopReason: 'error', + errorMessage: 'LLM exploded mid-stream', }); }); }); -describe("unknown routes", () => { - it("returns 404", async () => { - const res = await request("GET", "/unknown"); +describe('unknown routes', () => { + it('returns 404', async () => { + const res = await request('GET', '/unknown'); expect(res.status).toBe(404); }); }); diff --git a/packages/knative-server/test/solve-envelope.test.ts b/packages/knative-server/test/solve-envelope.test.ts index 1099ae6..632a4a8 100644 --- a/packages/knative-server/test/solve-envelope.test.ts +++ b/packages/knative-server/test/solve-envelope.test.ts @@ -1,21 +1,43 @@ -import { describe, it, expect } from "vitest"; -import { isSolveEnvelope, isRunEnvelope } from "../src/server.js"; +import { describe, it, expect } from 'vitest'; +import { isSolveEnvelope, isRunEnvelope } from '../src/server.js'; -describe("isSolveEnvelope", () => { - const ok = { sessionId: "s", kind: "solve", problemStatement: "fix it", repoUrl: "git://x/r.git", ref: "work" }; - it("accepts a well-formed solve envelope", () => { expect(isSolveEnvelope(ok)).toBe(true); }); - it("rejects a solve envelope missing repoUrl/ref/problemStatement", () => { - expect(isSolveEnvelope({ sessionId: "s", kind: "solve" })).toBe(false); +describe('isSolveEnvelope', () => { + const ok = { + sessionId: 's', + kind: 'solve', + problemStatement: 'fix it', + repoUrl: 'git://x/r.git', + ref: 'work', + }; + it('accepts a well-formed solve envelope', () => { + expect(isSolveEnvelope(ok)).toBe(true); }); - it("rejects a converge envelope (no kind:solve)", () => { - expect(isSolveEnvelope({ sessionId: "s", item: { item_id: "i", file: "f", pattern: "p" } })).toBe(false); + it('rejects a solve envelope missing repoUrl/ref/problemStatement', () => { + expect(isSolveEnvelope({ sessionId: 's', kind: 'solve' })).toBe(false); + }); + it('rejects a converge envelope (no kind:solve)', () => { + expect( + isSolveEnvelope({ sessionId: 's', item: { item_id: 'i', file: 'f', pattern: 'p' } }), + ).toBe(false); }); }); -describe("isRunEnvelope", () => { - it("accepts either a converge or a solve envelope", () => { - expect(isRunEnvelope({ sessionId: "s", item: { item_id: "i", file: "f", pattern: "p" } })).toBe(true); - expect(isRunEnvelope({ sessionId: "s", kind: "solve", problemStatement: "x", repoUrl: "g", ref: "r" })).toBe(true); +describe('isRunEnvelope', () => { + it('accepts either a converge or a solve envelope', () => { + expect(isRunEnvelope({ sessionId: 's', item: { item_id: 'i', file: 'f', pattern: 'p' } })).toBe( + true, + ); + expect( + isRunEnvelope({ + sessionId: 's', + kind: 'solve', + problemStatement: 'x', + repoUrl: 'g', + ref: 'r', + }), + ).toBe(true); + }); + it('rejects junk', () => { + expect(isRunEnvelope({ foo: 1 })).toBe(false); }); - it("rejects junk", () => { expect(isRunEnvelope({ foo: 1 })).toBe(false); }); }); diff --git a/packages/knative-server/test/swebench-sandbox-pool.test.ts b/packages/knative-server/test/swebench-sandbox-pool.test.ts index d2bab80..e924a4e 100644 --- a/packages/knative-server/test/swebench-sandbox-pool.test.ts +++ b/packages/knative-server/test/swebench-sandbox-pool.test.ts @@ -42,9 +42,12 @@ describe('swebench-sandbox-pool manifest', () => { } }); - it.each(sandboxes.map((s, i) => [i, s]))('sandbox %s: carries the CR-level app=sandbox label', (_i, sandbox: any) => { - expect(sandbox.metadata?.labels?.app).toBe('sandbox'); - }); + it.each(sandboxes.map((s, i) => [i, s]))( + 'sandbox %s: carries the CR-level app=sandbox label', + (_i, sandbox: any) => { + expect(sandbox.metadata?.labels?.app).toBe('sandbox'); + }, + ); it.each(sandboxes.map((s, i) => [i, s]))( 'sandbox %s: podTemplate pool-discovery label is exactly swebench (not default)', @@ -55,48 +58,73 @@ describe('swebench-sandbox-pool manifest', () => { }, ); - it.each(sandboxes.map((s, i) => [i, s]))('sandbox %s: container uses the Task-3 baked internal-registry image', (_i, sandbox: any) => { - const containers = sandbox.spec?.podTemplate?.spec?.containers ?? []; - expect(containers[0]?.image).toBe(SWEBENCH_IMAGE); - expect(containers[0]?.imagePullPolicy).toBe('IfNotPresent'); - }); + it.each(sandboxes.map((s, i) => [i, s]))( + 'sandbox %s: container uses the Task-3 baked internal-registry image', + (_i, sandbox: any) => { + const containers = sandbox.spec?.podTemplate?.spec?.containers ?? []; + expect(containers[0]?.image).toBe(SWEBENCH_IMAGE); + expect(containers[0]?.imagePullPolicy).toBe('IfNotPresent'); + }, + ); - it.each(sandboxes.map((s, i) => [i, s]))('sandbox %s: container command is exactly sleep infinity (no apk/startup install)', (_i, sandbox: any) => { - const containers = sandbox.spec?.podTemplate?.spec?.containers ?? []; - expect(containers[0]?.command).toEqual(['sleep', 'infinity']); - }); + it.each(sandboxes.map((s, i) => [i, s]))( + 'sandbox %s: container command is exactly sleep infinity (no apk/startup install)', + (_i, sandbox: any) => { + const containers = sandbox.spec?.podTemplate?.spec?.containers ?? []; + expect(containers[0]?.command).toEqual(['sleep', 'infinity']); + }, + ); - it.each(sandboxes.map((s, i) => [i, s]))('sandbox %s: container workingDir and workspace volumeMount are /workspace', (_i, sandbox: any) => { - const containers = sandbox.spec?.podTemplate?.spec?.containers ?? []; - const container = containers[0] ?? {}; - expect(container.workingDir).toBe('/workspace'); - const workspaceMount = (container.volumeMounts ?? []).find((m: any) => m.name === 'workspace'); - expect(workspaceMount?.mountPath).toBe('/workspace'); - }); + it.each(sandboxes.map((s, i) => [i, s]))( + 'sandbox %s: container workingDir and workspace volumeMount are /workspace', + (_i, sandbox: any) => { + const containers = sandbox.spec?.podTemplate?.spec?.containers ?? []; + const container = containers[0] ?? {}; + expect(container.workingDir).toBe('/workspace'); + const workspaceMount = (container.volumeMounts ?? []).find( + (m: any) => m.name === 'workspace', + ); + expect(workspaceMount?.mountPath).toBe('/workspace'); + }, + ); - it.each(sandboxes.map((s, i) => [i, s]))('sandbox %s: volumeClaimTemplate requests a 50Gi RWO PVC named workspace', (_i, sandbox: any) => { - const vct = (sandbox.spec?.volumeClaimTemplates ?? [])[0] ?? {}; - expect(vct.metadata?.name).toBe('workspace'); - expect(vct.spec?.accessModes).toEqual(['ReadWriteOnce']); - expect(vct.spec?.resources?.requests?.storage).toBe('50Gi'); - }); + it.each(sandboxes.map((s, i) => [i, s]))( + 'sandbox %s: volumeClaimTemplate requests a 50Gi RWO PVC named workspace', + (_i, sandbox: any) => { + const vct = (sandbox.spec?.volumeClaimTemplates ?? [])[0] ?? {}; + expect(vct.metadata?.name).toBe('workspace'); + expect(vct.spec?.accessModes).toEqual(['ReadWriteOnce']); + expect(vct.spec?.resources?.requests?.storage).toBe('50Gi'); + }, + ); - it.each(sandboxes.map((s, i) => [i, s]))('sandbox %s: podTemplate uses the serverless-harness-sandbox SA', (_i, sandbox: any) => { - expect(sandbox.spec?.podTemplate?.spec?.serviceAccountName).toBe('serverless-harness-sandbox'); - }); + it.each(sandboxes.map((s, i) => [i, s]))( + 'sandbox %s: podTemplate uses the serverless-harness-sandbox SA', + (_i, sandbox: any) => { + expect(sandbox.spec?.podTemplate?.spec?.serviceAccountName).toBe( + 'serverless-harness-sandbox', + ); + }, + ); - it.each(sandboxes.map((s, i) => [i, s]))('sandbox %s: pod-level securityContext is OCP nonroot', (_i, sandbox: any) => { - const podSecurityContext = sandbox.spec?.podTemplate?.spec?.securityContext ?? {}; - expect(podSecurityContext.runAsUser).toBe(65532); - expect(podSecurityContext.runAsNonRoot).toBe(true); - expect(podSecurityContext.fsGroup).toBe(65532); - expect(podSecurityContext.seccompProfile?.type).toBe('RuntimeDefault'); - }); + it.each(sandboxes.map((s, i) => [i, s]))( + 'sandbox %s: pod-level securityContext is OCP nonroot', + (_i, sandbox: any) => { + const podSecurityContext = sandbox.spec?.podTemplate?.spec?.securityContext ?? {}; + expect(podSecurityContext.runAsUser).toBe(65532); + expect(podSecurityContext.runAsNonRoot).toBe(true); + expect(podSecurityContext.fsGroup).toBe(65532); + expect(podSecurityContext.seccompProfile?.type).toBe('RuntimeDefault'); + }, + ); - it.each(sandboxes.map((s, i) => [i, s]))('sandbox %s: container-level securityContext drops all capabilities', (_i, sandbox: any) => { - const containers = sandbox.spec?.podTemplate?.spec?.containers ?? []; - const containerSecurityContext = containers[0]?.securityContext ?? {}; - expect(containerSecurityContext.allowPrivilegeEscalation).toBe(false); - expect(containerSecurityContext.capabilities?.drop).toContain('ALL'); - }); + it.each(sandboxes.map((s, i) => [i, s]))( + 'sandbox %s: container-level securityContext drops all capabilities', + (_i, sandbox: any) => { + const containers = sandbox.spec?.podTemplate?.spec?.containers ?? []; + const containerSecurityContext = containers[0]?.securityContext ?? {}; + expect(containerSecurityContext.allowPrivilegeEscalation).toBe(false); + expect(containerSecurityContext.capabilities?.drop).toContain('ALL'); + }, + ); }); diff --git a/packages/knative-server/test/worker-deployment.test.ts b/packages/knative-server/test/worker-deployment.test.ts index b2921b3..9477cf3 100644 --- a/packages/knative-server/test/worker-deployment.test.ts +++ b/packages/knative-server/test/worker-deployment.test.ts @@ -1,91 +1,101 @@ -import { readFileSync } from "node:fs"; -import { fileURLToPath } from "node:url"; -import { dirname, resolve } from "node:path"; -import { describe, expect, it } from "vitest"; -import { parse, parseAllDocuments } from "yaml"; +import { readFileSync } from 'node:fs'; +import { fileURLToPath } from 'node:url'; +import { dirname, resolve } from 'node:path'; +import { describe, expect, it } from 'vitest'; +import { parse, parseAllDocuments } from 'yaml'; type EnvVar = { name: string; value?: string }; -const REPO_ROOT = resolve(dirname(fileURLToPath(import.meta.url)), "../../.."); -const DEPLOY = resolve(REPO_ROOT, "deploy/knative"); -const WORKER = resolve(REPO_ROOT, "remote-worker"); +const REPO_ROOT = resolve(dirname(fileURLToPath(import.meta.url)), '../../..'); +const DEPLOY = resolve(REPO_ROOT, 'deploy/knative'); +const WORKER = resolve(REPO_ROOT, 'remote-worker'); -const example = () => parse(readFileSync(resolve(DEPLOY, "worker-example.yaml"), "utf8")); -const template = () => parse(readFileSync(resolve(WORKER, "worker-deployment.yaml"), "utf8")); +const example = () => parse(readFileSync(resolve(DEPLOY, 'worker-example.yaml'), 'utf8')); +const template = () => parse(readFileSync(resolve(WORKER, 'worker-deployment.yaml'), 'utf8')); const envOf = (dep: any): EnvVar[] => dep.spec.template.spec.containers[0].env; const get = (dep: any, name: string) => envOf(dep).find((e) => e.name === name)?.value; // Loud-throw readers for the memory/BufferCap coupling: a reformatted Go constant or a // malformed manifest value must fail the extraction itself, never limp through as NaN. const readBufferCapMiB = () => { - const runnerGo = readFileSync(resolve(WORKER, "internal/exec/runner.go"), "utf8"); + const runnerGo = readFileSync(resolve(WORKER, 'internal/exec/runner.go'), 'utf8'); const match = /BufferCap = (\d+) \* 1024 \* 1024/.exec(runnerGo); - if (!match) throw new Error("could not find `BufferCap = N * 1024 * 1024` in runner.go — constant reformatted?"); + if (!match) + throw new Error( + 'could not find `BufferCap = N * 1024 * 1024` in runner.go — constant reformatted?', + ); return Number(match[1]); }; const readDefaultConcurrency = () => { - const loopGo = readFileSync(resolve(WORKER, "internal/session/loop.go"), "utf8"); + const loopGo = readFileSync(resolve(WORKER, 'internal/session/loop.go'), 'utf8'); const match = /DefaultConcurrency = (\d+)/.exec(loopGo); - if (!match) throw new Error("could not find `DefaultConcurrency = N` in loop.go — constant reformatted?"); + if (!match) + throw new Error('could not find `DefaultConcurrency = N` in loop.go — constant reformatted?'); return Number(match[1]); }; const readLimitMiB = () => { const limit = template().spec.template.spec.containers[0].resources.limits.memory; const match = /^(\d+)Mi$/.exec(limit); - if (!match) throw new Error(`resources.limits.memory "${limit}" is not in the expected "Mi" form`); + if (!match) + throw new Error(`resources.limits.memory "${limit}" is not in the expected "Mi" form`); return Number(match[1]); }; -describe("worker-example.yaml (the third-party copy-and-edit surface)", () => { - it("is a single-replica Deployment with matching selector and labels", () => { +describe('worker-example.yaml (the third-party copy-and-edit surface)', () => { + it('is a single-replica Deployment with matching selector and labels', () => { const dep = example(); - expect(dep.kind).toBe("Deployment"); + expect(dep.kind).toBe('Deployment'); // One worker per SANDBOX_ID: the relay rejects a second live Attach for the same id, // so replicas > 1 would leave every extra pod crash-looping on a rejected Attach. expect(dep.spec.replicas).toBe(1); expect(dep.spec.selector.matchLabels.app).toBe(dep.spec.template.metadata.labels.app); }); - it("carries the three env vars a worker cannot start without", () => { + it('carries the three env vars a worker cannot start without', () => { const dep = example(); - for (const k of ["SANDBOX_ID", "RELAY_ADDR", "SANDBOX_TOKEN"]) { - expect(get(dep, k), `${k} is read at startup by remote-worker/cmd/worker/main.go`).toBeTruthy(); + for (const k of ['SANDBOX_ID', 'RELAY_ADDR', 'SANDBOX_TOKEN']) { + expect( + get(dep, k), + `${k} is read at startup by remote-worker/cmd/worker/main.go`, + ).toBeTruthy(); } }); - it("points RELAY_ADDR at the port the relay Service actually publishes", () => { + it('points RELAY_ADDR at the port the relay Service actually publishes', () => { // A drifted port here is the failure mode with the worst diagnostics: the worker // dials, gets connection-refused, backs off, and never appears in presence. - const relayDocs = parseAllDocuments(readFileSync(resolve(DEPLOY, "relay-deployment.yaml"), "utf8")).map((d) => d.toJS()); - const svc = relayDocs.find((o) => o.kind === "Service"); + const relayDocs = parseAllDocuments( + readFileSync(resolve(DEPLOY, 'relay-deployment.yaml'), 'utf8'), + ).map((d) => d.toJS()); + const svc = relayDocs.find((o) => o.kind === 'Service'); const port = svc.spec.ports[0].port; - expect(get(example(), "RELAY_ADDR")).toContain(`:${port}`); + expect(get(example(), 'RELAY_ADDR')).toContain(`:${port}`); }); - it("is restricted-v2 compatible: non-root, no privilege escalation, all caps dropped", () => { + it('is restricted-v2 compatible: non-root, no privilege escalation, all caps dropped', () => { const sc = example().spec.template.spec.containers[0].securityContext; expect(sc.runAsNonRoot).toBe(true); expect(sc.allowPrivilegeEscalation).toBe(false); - expect(sc.capabilities.drop).toContain("ALL"); + expect(sc.capabilities.drop).toContain('ALL'); // OpenShift assigns a UID from the namespace range; a hardcoded one breaks the // copy-and-edit path for anyone whose image does not use that exact UID. - expect(sc.runAsUser, "worker-example.yaml must not pin runAsUser").toBeUndefined(); + expect(sc.runAsUser, 'worker-example.yaml must not pin runAsUser').toBeUndefined(); }); }); -describe("remote-worker/worker-deployment.yaml (the sed-filled gate template)", () => { - it("keeps every placeholder deploy-incluster.sh substitutes", () => { - const raw = readFileSync(resolve(WORKER, "worker-deployment.yaml"), "utf8"); +describe('remote-worker/worker-deployment.yaml (the sed-filled gate template)', () => { + it('keeps every placeholder deploy-incluster.sh substitutes', () => { + const raw = readFileSync(resolve(WORKER, 'worker-deployment.yaml'), 'utf8'); // deploy-incluster.sh seds these by exact string; a rename here fails silently and // ships a pod with a literal __IMAGE__ reference. - for (const p of ["__NS__", "__IMAGE__", "__SANDBOX_ID__", "__TOKEN__"]) { + for (const p of ['__NS__', '__IMAGE__', '__SANDBOX_ID__', '__TOKEN__']) { expect(raw, `${p} is substituted by remote-worker/deploy-incluster.sh`).toContain(p); } }); - it("sets a memory limit that covers the worst-case buffering of whichever concurrency actually runs", () => { + it('sets a memory limit that covers the worst-case buffering of whichever concurrency actually runs', () => { // runner.go's BufferCap is a PER-STREAM cap on non-streaming execs, and // Exec.streaming=false is the proto3 default (relay-supplied), so a buggy relay // reaches the worst case with no privilege. Both files carry this arithmetic in a @@ -99,7 +109,7 @@ describe("remote-worker/worker-deployment.yaml (the sed-filled gate template)", // env var exists. const bufferCapMiB = readBufferCapMiB(); const limitMiB = readLimitMiB(); - const override = get(template(), "WORKER_MAX_CONCURRENT"); + const override = get(template(), 'WORKER_MAX_CONCURRENT'); let concurrency: number; let source: string; diff --git a/packages/knative-server/test/workload-route.test.ts b/packages/knative-server/test/workload-route.test.ts index 11a0697..b9ddeb6 100644 --- a/packages/knative-server/test/workload-route.test.ts +++ b/packages/knative-server/test/workload-route.test.ts @@ -1,44 +1,50 @@ -import { afterEach, beforeEach, describe, expect, it, vi } from "vitest"; - -const { records, runLeaf, configured, createWorkload, getWorkload, deleteWorkload } = vi.hoisted(() => ({ - records: new Map(), - runLeaf: vi.fn(), - configured: vi.fn(), - createWorkload: vi.fn(), - getWorkload: vi.fn(), - deleteWorkload: vi.fn(), -})); -vi.mock("@sh/harness/leaf-result-store", async (orig) => { - const actual = await orig(); +import { afterEach, beforeEach, describe, expect, it, vi } from 'vitest'; + +const { records, runLeaf, configured, createWorkload, getWorkload, deleteWorkload } = vi.hoisted( + () => ({ + records: new Map(), + runLeaf: vi.fn(), + configured: vi.fn(), + createWorkload: vi.fn(), + getWorkload: vi.fn(), + deleteWorkload: vi.fn(), + }), +); +vi.mock('@sh/harness/leaf-result-store', async (orig) => { + const actual = await orig(); class FakeStore { - async set(key: string, value: string) { records.set(key, value); } - async get(key: string) { return records.get(key) ?? null; } + async set(key: string, value: string) { + records.set(key, value); + } + async get(key: string) { + return records.get(key) ?? null; + } } return { ...actual, RedisResultStore: FakeStore }; }); -vi.mock("@sh/harness/run-leaf", () => ({ +vi.mock('@sh/harness/run-leaf', () => ({ runLeaf: (...args: any[]) => runLeaf(...args), validateItem: (item: any) => item, leafSessionId: (env: any) => env.sessionId, })); -vi.mock("../src/context-service.js", () => ({ +vi.mock('../src/context-service.js', () => ({ contextServiceConfigured: configured, createWorkload, getWorkload, deleteWorkload, })); -import { startServer } from "../src/server.js"; +import { startServer } from '../src/server.js'; const record = { - workloadId: "demo-workload", - status: "ready", + workloadId: 'demo-workload', + status: 'ready', replicas: 2, readyReplicas: 2, - sandboxSelector: "context.rossoctl.io/pool=demo-workload", - workspace: { size: "1Gi", accessMode: "ReadWriteMany", storageClass: "ibm-scale-csi" }, + sandboxSelector: 'context.rossoctl.io/pool=demo-workload', + workspace: { size: '1Gi', accessMode: 'ReadWriteMany', storageClass: 'ibm-scale-csi' }, }; let server: ReturnType; @@ -60,20 +66,23 @@ afterEach(() => server.close()); async function json(method: string, path: string, body?: unknown) { const response = await fetch(base + path, { method, - headers: { "content-type": "application/json" }, + headers: { 'content-type': 'application/json' }, ...(body === undefined ? {} : { body: JSON.stringify(body) }), }); return { status: response.status, body: await response.json().catch(() => null) }; } -describe("optional workload lifecycle", () => { - it("leaves ordinary runs unchanged when Context Service is disabled", async () => { +describe('optional workload lifecycle', () => { + it('leaves ordinary runs unchanged when Context Service is disabled', async () => { configured.mockReturnValue(false); - runLeaf.mockResolvedValue({ status: "done", verdict: { item_id: "i", verdict: "CLEAR", reason: "ok" } }); + runLeaf.mockResolvedValue({ + status: 'done', + verdict: { item_id: 'i', verdict: 'CLEAR', reason: 'ok' }, + }); - const response = await json("POST", "/runs", { - sessionId: "run/i", - item: { item_id: "i", file: "f", pattern: "p" }, + const response = await json('POST', '/runs', { + sessionId: 'run/i', + item: { item_id: 'i', file: 'f', pattern: 'p' }, }); expect(response.status).toBe(200); @@ -83,61 +92,68 @@ describe("optional workload lifecycle", () => { ); }); - it("reports that workload allocation is disabled when Context Service is not configured", async () => { + it('reports that workload allocation is disabled when Context Service is not configured', async () => { configured.mockReturnValue(false); - expect(await json("POST", "/workloads", { name: "demo-workload" })).toEqual({ + expect(await json('POST', '/workloads', { name: 'demo-workload' })).toEqual({ status: 501, - body: { error: "context_service_not_configured" }, + body: { error: 'context_service_not_configured' }, }); expect(createWorkload).not.toHaveBeenCalled(); }); - it("creates a workload through Context Service", async () => { - const response = await json("POST", "/workloads", { - name: "demo-workload", sandboxes: 2, - workspace: { shared: true, storageClass: "ibm-scale-csi" }, + it('creates a workload through Context Service', async () => { + const response = await json('POST', '/workloads', { + name: 'demo-workload', + sandboxes: 2, + workspace: { shared: true, storageClass: 'ibm-scale-csi' }, }); expect(response).toEqual({ status: 201, body: record }); - expect(createWorkload).toHaveBeenCalledWith("demo-workload", expect.objectContaining({ sandboxes: 2 })); + expect(createWorkload).toHaveBeenCalledWith( + 'demo-workload', + expect.objectContaining({ sandboxes: 2 }), + ); }); - it("does not expose Context Service errors to callers", async () => { - const log = vi.spyOn(console, "error").mockImplementation(() => undefined); - createWorkload.mockRejectedValueOnce(new Error("internal upstream detail")); + it('does not expose Context Service errors to callers', async () => { + const log = vi.spyOn(console, 'error').mockImplementation(() => undefined); + createWorkload.mockRejectedValueOnce(new Error('internal upstream detail')); - expect(await json("POST", "/workloads", { name: "demo-workload" })).toEqual({ + expect(await json('POST', '/workloads', { name: 'demo-workload' })).toEqual({ status: 502, - body: { error: "context_service_error" }, + body: { error: 'context_service_error' }, }); - expect(log).toHaveBeenCalledWith("Context Service create failed:", expect.any(Error)); + expect(log).toHaveBeenCalledWith('Context Service create failed:', expect.any(Error)); log.mockRestore(); }); - it("routes a run through its workload pool", async () => { - await json("POST", "/workloads", { name: "demo-workload" }); - runLeaf.mockResolvedValue({ status: "done", verdict: { item_id: "i", verdict: "CLEAR", reason: "ok" } }); - const response = await json("POST", "/runs", { - workloadId: "demo-workload", - sessionId: "run/i", - item: { item_id: "i", file: "f", pattern: "p" }, + it('routes a run through its workload pool', async () => { + await json('POST', '/workloads', { name: 'demo-workload' }); + runLeaf.mockResolvedValue({ + status: 'done', + verdict: { item_id: 'i', verdict: 'CLEAR', reason: 'ok' }, + }); + const response = await json('POST', '/runs', { + workloadId: 'demo-workload', + sessionId: 'run/i', + item: { item_id: 'i', file: 'f', pattern: 'p' }, }); expect(response.status).toBe(200); expect(runLeaf).toHaveBeenCalledWith( - expect.objectContaining({ sandboxPoolSelector: "context.rossoctl.io/pool=demo-workload" }), + expect.objectContaining({ sandboxPoolSelector: 'context.rossoctl.io/pool=demo-workload' }), expect.any(Object), ); }); - it("gates a prompt leaf on its workload but ignores the pool selector (ADR 0028)", async () => { - await json("POST", "/workloads", { name: "demo-workload" }); - const warn = vi.spyOn(console, "warn").mockImplementation(() => undefined); - runLeaf.mockResolvedValue({ status: "responded", text: "a summary" }); - const response = await json("POST", "/runs", { - workloadId: "demo-workload", - sessionId: "run/p1", - kind: "prompt", - prompt: "Summarize the repo.", - item: { item_id: "i", file: "f", pattern: "p" }, + it('gates a prompt leaf on its workload but ignores the pool selector (ADR 0028)', async () => { + await json('POST', '/workloads', { name: 'demo-workload' }); + const warn = vi.spyOn(console, 'warn').mockImplementation(() => undefined); + runLeaf.mockResolvedValue({ status: 'responded', text: 'a summary' }); + const response = await json('POST', '/runs', { + workloadId: 'demo-workload', + sessionId: 'run/p1', + kind: 'prompt', + prompt: 'Summarize the repo.', + item: { item_id: 'i', file: 'f', pattern: 'p' }, }); expect(response.status).toBe(200); expect(runLeaf).toHaveBeenCalledWith( @@ -148,20 +164,20 @@ describe("optional workload lifecycle", () => { warn.mockRestore(); }); - it("deletes the workload through Context Service", async () => { - await json("POST", "/workloads", { name: "demo-workload" }); - const response = await fetch(base + "/workloads/demo-workload", { method: "DELETE" }); + it('deletes the workload through Context Service', async () => { + await json('POST', '/workloads', { name: 'demo-workload' }); + const response = await fetch(base + '/workloads/demo-workload', { method: 'DELETE' }); expect(response.status).toBe(204); - expect(deleteWorkload).toHaveBeenCalledWith("demo-workload"); + expect(deleteWorkload).toHaveBeenCalledWith('demo-workload'); }); - it("rejects a run for an unknown workload", async () => { - const response = await json("POST", "/runs", { - workloadId: "missing", - sessionId: "run/i", - item: { item_id: "i", file: "f", pattern: "p" }, + it('rejects a run for an unknown workload', async () => { + const response = await json('POST', '/runs', { + workloadId: 'missing', + sessionId: 'run/i', + item: { item_id: 'i', file: 'f', pattern: 'p' }, }); - expect(response).toEqual({ status: 404, body: { error: "workload_not_found" } }); + expect(response).toEqual({ status: 404, body: { error: 'workload_not_found' } }); expect(runLeaf).not.toHaveBeenCalled(); }); }); diff --git a/packages/knative-server/vitest.config.ts b/packages/knative-server/vitest.config.ts index c794660..e1e28f8 100644 --- a/packages/knative-server/vitest.config.ts +++ b/packages/knative-server/vitest.config.ts @@ -1,5 +1,5 @@ -import { defineConfig } from "vitest/config"; +import { defineConfig } from 'vitest/config'; export default defineConfig({ - test: { include: ["test/**/*.test.ts"] }, + test: { include: ['test/**/*.test.ts'] }, }); diff --git a/packages/sandbox-relay/package.json b/packages/sandbox-relay/package.json index bf2c8fa..19e6a60 100644 --- a/packages/sandbox-relay/package.json +++ b/packages/sandbox-relay/package.json @@ -2,7 +2,9 @@ "name": "@sh/sandbox-relay", "type": "module", "version": "0.0.0", - "exports": { ".": "./src/index.ts" }, + "exports": { + ".": "./src/index.ts" + }, "scripts": { "start": "node --import tsx src/main.ts", "test": "vitest run" diff --git a/packages/sandbox-relay/src/index.ts b/packages/sandbox-relay/src/index.ts index 0d5332d..ff59bcc 100644 --- a/packages/sandbox-relay/src/index.ts +++ b/packages/sandbox-relay/src/index.ts @@ -1,2 +1,2 @@ -export { createRelay, type Relay, type RelayDeps, type AttachStream } from "./relay.js"; -export { buildServer, startRelay } from "./main.js"; +export { createRelay, type Relay, type RelayDeps, type AttachStream } from './relay.js'; +export { buildServer, startRelay } from './main.js'; diff --git a/packages/sandbox-relay/src/main.ts b/packages/sandbox-relay/src/main.ts index 3599b98..66f79ce 100644 --- a/packages/sandbox-relay/src/main.ts +++ b/packages/sandbox-relay/src/main.ts @@ -5,7 +5,7 @@ import { type ServerWritableStream, type ServerUnaryCall, type sendUnaryData, -} from "@grpc/grpc-js"; +} from '@grpc/grpc-js'; import { SandboxWorkerService, SandboxExecService, @@ -17,9 +17,9 @@ import { type ExecEvent, type AbortRequest, type AbortResponse, -} from "@sh/k8s-sandbox"; -import { RedisRecordStore } from "@sh/harness"; -import { createRelay, type RelayDeps, type AttachStream } from "./relay.js"; +} from '@sh/k8s-sandbox'; +import { RedisRecordStore } from '@sh/harness'; +import { createRelay, type RelayDeps, type AttachStream } from './relay.js'; export function buildServer(deps: RelayDeps): { server: Server } { const relay = createRelay(deps); @@ -30,7 +30,8 @@ export function buildServer(deps: RelayDeps): { server: Server } { // Metadata.get() returns MetadataValue[] (string | Buffer). The relay only // ever reads a bearer token (always sent as a string by well-behaved // clients), so the cast is safe here without widening relay.ts's contract. - attach: (call: ServerDuplexStream) => relay.onAttach(call as unknown as AttachStream), + attach: (call: ServerDuplexStream) => + relay.onAttach(call as unknown as AttachStream), }; server.addService(SandboxWorkerService, workerImpl); @@ -50,26 +51,36 @@ export function buildServer(deps: RelayDeps): { server: Server } { const req = call.request; const e = req.exec; if (!e) { - call.destroy(new Error("ExecRequest missing exec field")); + call.destroy(new Error('ExecRequest missing exec field')); return; } // Registered synchronously (before the loop's first await) so a // cancellation that races the very first event is never missed. const onCancelled = () => relay.routeAbort(req.sandboxId, e.reqId); - call.on("cancelled", onCancelled); + call.on('cancelled', onCancelled); try { - for await (const ev of relay.routeExec(req.sandboxId, e.reqId, e.command, e.stdin, e.timeoutS, e.streaming)) { + for await (const ev of relay.routeExec( + req.sandboxId, + e.reqId, + e.command, + e.stdin, + e.timeoutS, + e.streaming, + )) { call.write(ev); } call.end(); } catch (err) { call.destroy(err as Error); } finally { - call.removeListener("cancelled", onCancelled); + call.removeListener('cancelled', onCancelled); } }, - abort: (call: ServerUnaryCall, cb: sendUnaryData) => { + abort: ( + call: ServerUnaryCall, + cb: sendUnaryData, + ) => { relay.routeAbort(call.request.sandboxId, call.request.reqId); cb(null, {}); }, @@ -106,7 +117,9 @@ export async function startRelay( const { server } = buildServer(deps); const addr = `0.0.0.0:${opts.port ?? Number(process.env.SH_RELAY_PORT ?? 8443)}`; const port = await new Promise((resolve, reject) => - server.bindAsync(addr, ServerCredentials.createInsecure(), (err, p) => (err ? reject(err) : resolve(p))), + server.bindAsync(addr, ServerCredentials.createInsecure(), (err, p) => + err ? reject(err) : resolve(p), + ), ); return { port, shutdown: () => new Promise((r) => server.tryShutdown(() => r())) }; } diff --git a/packages/sandbox-relay/src/relay.ts b/packages/sandbox-relay/src/relay.ts index 830c92d..b1095f4 100644 --- a/packages/sandbox-relay/src/relay.ts +++ b/packages/sandbox-relay/src/relay.ts @@ -1,11 +1,11 @@ -import type { RecordStore, SandboxRecord } from "@sh/harness"; -import type { ExecEvent, ServerFrame, WorkerFrame } from "@sh/k8s-sandbox"; +import type { RecordStore, SandboxRecord } from '@sh/harness'; +import type { ExecEvent, ServerFrame, WorkerFrame } from '@sh/k8s-sandbox'; export interface AttachStream { metadata?: { get: (k: string) => string[] }; - on(event: "data", cb: (f: WorkerFrame) => void): unknown; - on(event: "end", cb: () => void): unknown; - on(event: "error", cb: (e: Error) => void): unknown; + on(event: 'data', cb: (f: WorkerFrame) => void): unknown; + on(event: 'end', cb: () => void): unknown; + on(event: 'error', cb: (e: Error) => void): unknown; write(f: ServerFrame): void; end(): void; } @@ -36,8 +36,8 @@ export interface Relay { } function bearer(md?: { get: (k: string) => string[] }): string | undefined { - const v = md?.get("authorization")?.[0]; - return v?.startsWith("Bearer ") ? v.slice(7) : undefined; + const v = md?.get('authorization')?.[0]; + return v?.startsWith('Bearer ') ? v.slice(7) : undefined; } export function createRelay(deps: RelayDeps): Relay { @@ -45,7 +45,7 @@ export function createRelay(deps: RelayDeps): Relay { function onAttach(stream: AttachStream): void { let sandboxId: string | undefined; - stream.on("data", (frame: WorkerFrame) => { + stream.on('data', (frame: WorkerFrame) => { if (frame.hello && !sandboxId) { const id = frame.hello.sandboxId; if (!deps.validateToken(bearer(stream.metadata), id)) { @@ -73,9 +73,9 @@ export function createRelay(deps: RelayDeps): Relay { // select-sandbox still leases against its own opts.cap. Wiring // capacityMax into leasing decisions is a later slice. capacityMax: frame.hello.capacityMax, - transport: "grpc", + transport: 'grpc', }; - void deps.records.put(rec).catch((e) => console.error("presence put failed", e)); + void deps.records.put(rec).catch((e) => console.error('presence put failed', e)); return; } // chunk/end/error frames are dispatched to the per-reqId sink registered by routeExec @@ -91,15 +91,17 @@ export function createRelay(deps: RelayDeps): Relay { const parked = sessions.get(sandboxId); if (parked) { for (const [reqId, sink] of parked.sinks) { - sink({ error: { reqId, message: "worker disconnected" } } as ExecEvent); + sink({ error: { reqId, message: 'worker disconnected' } } as ExecEvent); } } sessions.delete(sandboxId); - void deps.records.remove(sandboxId).catch((e) => console.error("presence remove failed", e)); + void deps.records + .remove(sandboxId) + .catch((e) => console.error('presence remove failed', e)); } }; - stream.on("end", teardown); - stream.on("error", teardown); + stream.on('end', teardown); + stream.on('error', teardown); } async function* routeExec( diff --git a/packages/sandbox-relay/test/main-default-token.test.ts b/packages/sandbox-relay/test/main-default-token.test.ts index c8f7770..2faef55 100644 --- a/packages/sandbox-relay/test/main-default-token.test.ts +++ b/packages/sandbox-relay/test/main-default-token.test.ts @@ -1,35 +1,38 @@ -import { describe, expect, it } from "vitest"; -import { makeDefaultValidateToken } from "../src/main.js"; +import { describe, expect, it } from 'vitest'; +import { makeDefaultValidateToken } from '../src/main.js'; -describe("makeDefaultValidateToken (fail-closed default auth)", () => { - it("rejects any token, including undefined, when no token env var is configured", () => { +describe('makeDefaultValidateToken (fail-closed default auth)', () => { + it('rejects any token, including undefined, when no token env var is configured', () => { const validate = makeDefaultValidateToken({}); - expect(validate(undefined, "sbx-1")).toBe(false); - expect(validate("", "sbx-1")).toBe(false); - expect(validate("anything", "sbx-1")).toBe(false); + expect(validate(undefined, 'sbx-1')).toBe(false); + expect(validate('', 'sbx-1')).toBe(false); + expect(validate('anything', 'sbx-1')).toBe(false); }); - it("SH_RELAY_TOKEN set: only an exact match is accepted", () => { - const validate = makeDefaultValidateToken({ SH_RELAY_TOKEN: "secret" }); - expect(validate("secret", "sbx-1")).toBe(true); - expect(validate("wrong", "sbx-1")).toBe(false); - expect(validate(undefined, "sbx-1")).toBe(false); - expect(validate("", "sbx-1")).toBe(false); + it('SH_RELAY_TOKEN set: only an exact match is accepted', () => { + const validate = makeDefaultValidateToken({ SH_RELAY_TOKEN: 'secret' }); + expect(validate('secret', 'sbx-1')).toBe(true); + expect(validate('wrong', 'sbx-1')).toBe(false); + expect(validate(undefined, 'sbx-1')).toBe(false); + expect(validate('', 'sbx-1')).toBe(false); }); - it("SH_RELAY_TOKEN_ per-sandbox override takes precedence for that sandbox", () => { - const validate = makeDefaultValidateToken({ SH_RELAY_TOKEN: "global", SH_RELAY_TOKEN_sbx1: "onlysbx1" }); - expect(validate("onlysbx1", "sbx1")).toBe(true); + it('SH_RELAY_TOKEN_ per-sandbox override takes precedence for that sandbox', () => { + const validate = makeDefaultValidateToken({ + SH_RELAY_TOKEN: 'global', + SH_RELAY_TOKEN_sbx1: 'onlysbx1', + }); + expect(validate('onlysbx1', 'sbx1')).toBe(true); // The global token must not authenticate a sandbox that has its own override. - expect(validate("global", "sbx1")).toBe(false); + expect(validate('global', 'sbx1')).toBe(false); // A different sandbox with no override falls back to the global token. - expect(validate("global", "sbx2")).toBe(true); + expect(validate('global', 'sbx2')).toBe(true); }); - it("per-sandbox override with no global token set still fails closed for other sandboxes", () => { - const validate = makeDefaultValidateToken({ SH_RELAY_TOKEN_sbx1: "onlysbx1" }); - expect(validate("onlysbx1", "sbx1")).toBe(true); - expect(validate(undefined, "sbx2")).toBe(false); - expect(validate("onlysbx1", "sbx2")).toBe(false); + it('per-sandbox override with no global token set still fails closed for other sandboxes', () => { + const validate = makeDefaultValidateToken({ SH_RELAY_TOKEN_sbx1: 'onlysbx1' }); + expect(validate('onlysbx1', 'sbx1')).toBe(true); + expect(validate(undefined, 'sbx2')).toBe(false); + expect(validate('onlysbx1', 'sbx2')).toBe(false); }); }); diff --git a/packages/sandbox-relay/test/main-wiring.test.ts b/packages/sandbox-relay/test/main-wiring.test.ts index d4616ba..c569d5f 100644 --- a/packages/sandbox-relay/test/main-wiring.test.ts +++ b/packages/sandbox-relay/test/main-wiring.test.ts @@ -1,20 +1,21 @@ -import { EventEmitter } from "node:events"; -import { describe, expect, it, vi } from "vitest"; -import { buildServer } from "../src/main.js"; -import type { RecordStore } from "@sh/harness"; +import { EventEmitter } from 'node:events'; +import { describe, expect, it, vi } from 'vitest'; +import { buildServer } from '../src/main.js'; +import type { RecordStore } from '@sh/harness'; const records: RecordStore = { put: async () => {}, remove: async () => {}, list: async () => [] }; /** Grabs the exact bound handler grpc-js registered for a full method path. */ function getHandler(server: unknown, path: string): (call: unknown) => unknown { - const handlers = (server as { handlers: Map unknown }> }).handlers; + const handlers = (server as { handlers: Map unknown }> }) + .handlers; const entry = handlers.get(path); if (!entry) throw new Error(`no handler registered for ${path}`); return entry.func; } -describe("relay server wiring", () => { - it("registers both gRPC services", () => { +describe('relay server wiring', () => { + it('registers both gRPC services', () => { const { server } = buildServer({ records, validateToken: () => true }); // grpc-js Server keeps registered handlers in a private `handlers` Map keyed // by full method path (e.g. "/sandbox.v1.SandboxWorker/Attach"). The brief's @@ -23,8 +24,8 @@ describe("relay server wiring", () => { const handlers = (server as unknown as { handlers: Map }).handlers; expect(handlers).toBeInstanceOf(Map); const names = [...handlers.keys()]; - expect(names.some((n) => n.includes("SandboxWorker"))).toBe(true); - expect(names.some((n) => n.includes("SandboxExec"))).toBe(true); + expect(names.some((n) => n.includes('SandboxWorker'))).toBe(true); + expect(names.some((n) => n.includes('SandboxExec'))).toBe(true); }); }); @@ -39,7 +40,7 @@ function fakeAttach() { s.metadata = { get: () => [] }; s.written = []; s.write = (f) => s.written.push(f); - s.end = () => s.emit("end"); + s.end = () => s.emit('end'); return s; } @@ -63,50 +64,72 @@ function fakeExecCall(request: unknown) { return c; } -describe("relay server exec cancellation wiring (via the real registered handler)", () => { - it("aborts the worker on client cancel and cleanly drains the generator", async () => { +describe('relay server exec cancellation wiring (via the real registered handler)', () => { + it('aborts the worker on client cancel and cleanly drains the generator', async () => { const { server } = buildServer({ records, validateToken: () => true }); - const attach = getHandler(server, "/sandbox.v1.SandboxWorker/Attach"); - const exec = getHandler(server, "/sandbox.v1.SandboxExec/Exec"); + const attach = getHandler(server, '/sandbox.v1.SandboxWorker/Attach'); + const exec = getHandler(server, '/sandbox.v1.SandboxExec/Exec'); const worker = fakeAttach(); attach(worker); - worker.emit("data", { - hello: { sandboxId: "sbx-1", labels: {}, capabilities: [], image: "", arch: "amd64", capacityMax: 1, trust: "trusted" }, + worker.emit('data', { + hello: { + sandboxId: 'sbx-1', + labels: {}, + capabilities: [], + image: '', + arch: 'amd64', + capacityMax: 1, + trust: 'trusted', + }, }); const call = fakeExecCall({ - sandboxId: "sbx-1", - exec: { reqId: 1, command: "sleep 100", stdin: new Uint8Array(), timeoutS: 0, streaming: true }, + sandboxId: 'sbx-1', + exec: { + reqId: 1, + command: 'sleep 100', + stdin: new Uint8Array(), + timeoutS: 0, + streaming: true, + }, }); exec(call); // routeExec has parked its sink and written ServerFrame{exec} to the worker. - await vi.waitFor(() => expect((worker.written.at(-1) as { exec?: { reqId: number } })?.exec?.reqId).toBe(1)); + await vi.waitFor(() => + expect((worker.written.at(-1) as { exec?: { reqId: number } })?.exec?.reqId).toBe(1), + ); // Harness (client) cancels its own call -- e.g. its deadline fired. - call.emit("cancelled"); + call.emit('cancelled'); // The handler must NOT just call .return() on the idling generator; it must // tell the worker to abort so the worker's own reply drives cleanup. - await vi.waitFor(() => expect((worker.written.at(-1) as { abort?: { reqId: number } })?.abort?.reqId).toBe(1)); + await vi.waitFor(() => + expect((worker.written.at(-1) as { abort?: { reqId: number } })?.abort?.reqId).toBe(1), + ); // Worker honors the abort with an error frame for that reqId. - worker.emit("data", { error: { reqId: 1, message: "aborted" } }); + worker.emit('data', { error: { reqId: 1, message: 'aborted' } }); // The generator yields that event and returns -- exec handler writes it and ends. - await vi.waitFor(() => expect((call.written.at(-1) as { error?: { message: string } })?.error?.message).toBe("aborted")); + await vi.waitFor(() => + expect((call.written.at(-1) as { error?: { message: string } })?.error?.message).toBe( + 'aborted', + ), + ); await vi.waitFor(() => expect(call.ended).toBe(true)); expect(call.destroyed).toBeUndefined(); }); - it("destroys the call when routeExec throws (e.g. absent sandbox)", async () => { + it('destroys the call when routeExec throws (e.g. absent sandbox)', async () => { const { server } = buildServer({ records, validateToken: () => true }); - const exec = getHandler(server, "/sandbox.v1.SandboxExec/Exec"); + const exec = getHandler(server, '/sandbox.v1.SandboxExec/Exec'); const call = fakeExecCall({ - sandboxId: "ghost", - exec: { reqId: 1, command: "x", stdin: new Uint8Array(), timeoutS: 0, streaming: true }, + sandboxId: 'ghost', + exec: { reqId: 1, command: 'x', stdin: new Uint8Array(), timeoutS: 0, streaming: true }, }); exec(call); diff --git a/packages/sandbox-relay/test/relay-attach.test.ts b/packages/sandbox-relay/test/relay-attach.test.ts index 9c89460..aaee169 100644 --- a/packages/sandbox-relay/test/relay-attach.test.ts +++ b/packages/sandbox-relay/test/relay-attach.test.ts @@ -1,7 +1,7 @@ -import { EventEmitter } from "node:events"; -import { describe, expect, it, vi } from "vitest"; -import { createRelay } from "../src/relay.js"; -import type { SandboxRecord, RecordStore } from "@sh/harness"; +import { EventEmitter } from 'node:events'; +import { describe, expect, it, vi } from 'vitest'; +import { createRelay } from '../src/relay.js'; +import type { SandboxRecord, RecordStore } from '@sh/harness'; function fakeRecords() { const map = new Map(); @@ -22,82 +22,90 @@ function fakeAttach(token?: string) { written: unknown[]; emitData: (f: unknown) => void; }; - s.metadata = { get: (k) => (k === "authorization" && token ? [`Bearer ${token}`] : []) }; + s.metadata = { get: (k) => (k === 'authorization' && token ? [`Bearer ${token}`] : []) }; s.written = []; s.write = (f) => s.written.push(f); - s.end = () => s.emit("end"); - s.emitData = (f) => s.emit("data", f); + s.end = () => s.emit('end'); + s.emitData = (f) => s.emit('data', f); return s; } const hello = (sandboxId: string) => ({ - hello: { sandboxId, labels: { team: "t1" }, capabilities: ["python3"], image: "img", arch: "amd64", capacityMax: 4, trust: "trusted" }, + hello: { + sandboxId, + labels: { team: 't1' }, + capabilities: ['python3'], + image: 'img', + arch: 'amd64', + capacityMax: 4, + trust: 'trusted', + }, }); -describe("relay Attach + presence", () => { - it("parks the stream and mirrors presence on Hello", async () => { +describe('relay Attach + presence', () => { + it('parks the stream and mirrors presence on Hello', async () => { const { store, map } = fakeRecords(); const relay = createRelay({ records: store, validateToken: () => true } as never); - const s = fakeAttach("good"); + const s = fakeAttach('good'); relay.onAttach(s as never); - s.emitData(hello("sbx-1")); - await vi.waitFor(() => expect(map.get("sbx-1")).toBeTruthy()); - expect(map.get("sbx-1")!.transport).toBe("grpc"); - expect(relay.parked()).toContain("sbx-1"); + s.emitData(hello('sbx-1')); + await vi.waitFor(() => expect(map.get('sbx-1')).toBeTruthy()); + expect(map.get('sbx-1')!.transport).toBe('grpc'); + expect(relay.parked()).toContain('sbx-1'); }); - it("removes presence when the stream closes", async () => { + it('removes presence when the stream closes', async () => { const { store, map } = fakeRecords(); const relay = createRelay({ records: store, validateToken: () => true } as never); - const s = fakeAttach("good"); + const s = fakeAttach('good'); relay.onAttach(s as never); - s.emitData(hello("sbx-1")); - await vi.waitFor(() => expect(map.get("sbx-1")).toBeTruthy()); + s.emitData(hello('sbx-1')); + await vi.waitFor(() => expect(map.get('sbx-1')).toBeTruthy()); s.end(); - await vi.waitFor(() => expect(map.get("sbx-1")).toBeUndefined()); - expect(relay.parked()).not.toContain("sbx-1"); + await vi.waitFor(() => expect(map.get('sbx-1')).toBeUndefined()); + expect(relay.parked()).not.toContain('sbx-1'); }); - it("rejects a bad token before parking (no presence written)", async () => { + it('rejects a bad token before parking (no presence written)', async () => { const { store, map } = fakeRecords(); const relay = createRelay({ records: store, validateToken: () => false } as never); - const s = fakeAttach("bad"); + const s = fakeAttach('bad'); relay.onAttach(s as never); - s.emitData(hello("sbx-1")); + s.emitData(hello('sbx-1')); await new Promise((r) => setTimeout(r, 10)); - expect(map.get("sbx-1")).toBeUndefined(); - expect(relay.parked()).not.toContain("sbx-1"); + expect(map.get('sbx-1')).toBeUndefined(); + expect(relay.parked()).not.toContain('sbx-1'); }); - it("rejects a duplicate Attach for an already-parked sandboxId without evicting the first session", async () => { + it('rejects a duplicate Attach for an already-parked sandboxId without evicting the first session', async () => { const { store, map } = fakeRecords(); const relay = createRelay({ records: store, validateToken: () => true } as never); - const s1 = fakeAttach("good"); + const s1 = fakeAttach('good'); relay.onAttach(s1 as never); - s1.emitData(hello("sbx-1")); - await vi.waitFor(() => expect(map.get("sbx-1")).toBeTruthy()); - const presenceAfterFirst = map.get("sbx-1"); + s1.emitData(hello('sbx-1')); + await vi.waitFor(() => expect(map.get('sbx-1')).toBeTruthy()); + const presenceAfterFirst = map.get('sbx-1'); // A second worker tries to claim the same sandboxId while worker-1 is still live. - const s2 = fakeAttach("good"); + const s2 = fakeAttach('good'); let s2Ended = false; s2.end = () => { s2Ended = true; - s2.emit("end"); + s2.emit('end'); }; relay.onAttach(s2 as never); - s2.emitData(hello("sbx-1")); + s2.emitData(hello('sbx-1')); // The duplicate is rejected: its stream is ended, no session replacement, presence untouched. expect(s2Ended).toBe(true); - expect(relay.parked()).toContain("sbx-1"); - expect(map.get("sbx-1")).toBe(presenceAfterFirst); + expect(relay.parked()).toContain('sbx-1'); + expect(map.get('sbx-1')).toBe(presenceAfterFirst); // Worker-1's teardown (its own "end") still removes the (only, original) session — // proof that worker-2's Hello never replaced it. s1.end(); - await vi.waitFor(() => expect(map.get("sbx-1")).toBeUndefined()); - expect(relay.parked()).not.toContain("sbx-1"); + await vi.waitFor(() => expect(map.get('sbx-1')).toBeUndefined()); + expect(relay.parked()).not.toContain('sbx-1'); }); }); diff --git a/packages/sandbox-relay/test/relay-exec.test.ts b/packages/sandbox-relay/test/relay-exec.test.ts index 6b35c69..593e9c2 100644 --- a/packages/sandbox-relay/test/relay-exec.test.ts +++ b/packages/sandbox-relay/test/relay-exec.test.ts @@ -1,68 +1,108 @@ -import { EventEmitter } from "node:events"; -import { describe, expect, it, vi } from "vitest"; -import { createRelay } from "../src/relay.js"; -import type { RecordStore } from "@sh/harness"; +import { EventEmitter } from 'node:events'; +import { describe, expect, it, vi } from 'vitest'; +import { createRelay } from '../src/relay.js'; +import type { RecordStore } from '@sh/harness'; const records: RecordStore = { put: async () => {}, remove: async () => {}, list: async () => [] }; function fakeAttach() { - const s = new EventEmitter() as EventEmitter & { metadata: { get: () => string[] }; write: (f: any) => void; end: () => void; written: any[]; emitData: (f: any) => void }; + const s = new EventEmitter() as EventEmitter & { + metadata: { get: () => string[] }; + write: (f: any) => void; + end: () => void; + written: any[]; + emitData: (f: any) => void; + }; s.metadata = { get: () => [] }; s.written = []; s.write = (f) => s.written.push(f); - s.end = () => s.emit("end"); - s.emitData = (f) => s.emit("data", f); + s.end = () => s.emit('end'); + s.emitData = (f) => s.emit('data', f); return s; } -describe("relay Exec/Abort routing", () => { +describe('relay Exec/Abort routing', () => { it("routes Exec to the worker and yields the worker's ExecEvents back", async () => { const relay = createRelay({ records, validateToken: () => true } as never); const s = fakeAttach(); relay.onAttach(s as never); - s.emitData({ hello: { sandboxId: "sbx-1", labels: {}, capabilities: [], image: "", arch: "amd64", capacityMax: 1, trust: "trusted" } }); + s.emitData({ + hello: { + sandboxId: 'sbx-1', + labels: {}, + capabilities: [], + image: '', + arch: 'amd64', + capacityMax: 1, + trust: 'trusted', + }, + }); const events: any[] = []; const pump = (async () => { - for await (const ev of relay.routeExec("sbx-1", 1, "echo hi", new Uint8Array(), 0, true)) events.push(ev); + for await (const ev of relay.routeExec('sbx-1', 1, 'echo hi', new Uint8Array(), 0, true)) + events.push(ev); })(); // The relay should have sent a ServerFrame{exec} to the worker. await vi.waitFor(() => expect(s.written.at(-1)?.exec?.reqId).toBe(1)); // Worker replies with a chunk then end. - s.emitData({ chunk: { reqId: 1, data: Buffer.from("hi"), stream: 1 } }); + s.emitData({ chunk: { reqId: 1, data: Buffer.from('hi'), stream: 1 } }); s.emitData({ end: { reqId: 1, exitCode: 0 } }); await pump; - expect(events.map((e) => e.chunk?.data && Buffer.from(e.chunk.data).toString()).filter(Boolean)).toContain("hi"); + expect( + events.map((e) => e.chunk?.data && Buffer.from(e.chunk.data).toString()).filter(Boolean), + ).toContain('hi'); expect(events.at(-1).end.exitCode).toBe(0); }); - it("Abort sends ServerFrame{abort} to the worker", async () => { + it('Abort sends ServerFrame{abort} to the worker', async () => { const relay = createRelay({ records, validateToken: () => true } as never); const s = fakeAttach(); relay.onAttach(s as never); - s.emitData({ hello: { sandboxId: "sbx-1", labels: {}, capabilities: [], image: "", arch: "amd64", capacityMax: 1, trust: "trusted" } }); - relay.routeAbort("sbx-1", 5); + s.emitData({ + hello: { + sandboxId: 'sbx-1', + labels: {}, + capabilities: [], + image: '', + arch: 'amd64', + capacityMax: 1, + trust: 'trusted', + }, + }); + relay.routeAbort('sbx-1', 5); expect(s.written.at(-1)?.abort?.reqId).toBe(5); }); - it("Exec for an absent sandboxId throws", async () => { + it('Exec for an absent sandboxId throws', async () => { const relay = createRelay({ records, validateToken: () => true } as never); await expect(async () => { - for await (const _ of relay.routeExec("ghost", 1, "x", new Uint8Array(), 0, true)) void _; + for await (const _ of relay.routeExec('ghost', 1, 'x', new Uint8Array(), 0, true)) void _; }).rejects.toThrow(/no live worker/); }); - it("worker disconnect mid-exec fails the in-flight routeExec generator fast", async () => { + it('worker disconnect mid-exec fails the in-flight routeExec generator fast', async () => { const relay = createRelay({ records, validateToken: () => true } as never); const s = fakeAttach(); relay.onAttach(s as never); - s.emitData({ hello: { sandboxId: "sbx-1", labels: {}, capabilities: [], image: "", arch: "amd64", capacityMax: 1, trust: "trusted" } }); + s.emitData({ + hello: { + sandboxId: 'sbx-1', + labels: {}, + capabilities: [], + image: '', + arch: 'amd64', + capacityMax: 1, + trust: 'trusted', + }, + }); const events: any[] = []; let finished = false; const pump = (async () => { - for await (const ev of relay.routeExec("sbx-1", 1, "sleep 100", new Uint8Array(), 0, true)) events.push(ev); + for await (const ev of relay.routeExec('sbx-1', 1, 'sleep 100', new Uint8Array(), 0, true)) + events.push(ev); finished = true; })(); @@ -77,23 +117,34 @@ describe("relay Exec/Abort routing", () => { // await, so this would hang until the test's own timeout. await vi.waitFor(() => expect(finished).toBe(true)); - expect(events.at(-1)?.error?.message).toBe("worker disconnected"); + expect(events.at(-1)?.error?.message).toBe('worker disconnected'); // The sink must have been cleaned up (routeExec's finally ran) -- no leak. - expect(relay.parked()).not.toContain("sbx-1"); + expect(relay.parked()).not.toContain('sbx-1'); }); - it("refuses a second in-flight exec with the same req_id instead of overwriting the first", async () => { + it('refuses a second in-flight exec with the same req_id instead of overwriting the first', async () => { // Sinks are keyed by req_id per parked session, so an overwrite silently detaches // the first caller (it then hangs to its own deadline) and hands its frames to the // second. Failing loudly is strictly better than cross-talk (#179). const relay = createRelay({ records, validateToken: () => true } as never); const s = fakeAttach(); relay.onAttach(s as never); - s.emitData({ hello: { sandboxId: "sbx-1", labels: {}, capabilities: [], image: "", arch: "amd64", capacityMax: 1, trust: "trusted" } }); + s.emitData({ + hello: { + sandboxId: 'sbx-1', + labels: {}, + capabilities: [], + image: '', + arch: 'amd64', + capacityMax: 1, + trust: 'trusted', + }, + }); const events: any[] = []; const pump = (async () => { - for await (const ev of relay.routeExec("sbx-1", 7, "sleep 5", new Uint8Array(), 0, true)) events.push(ev); + for await (const ev of relay.routeExec('sbx-1', 7, 'sleep 5', new Uint8Array(), 0, true)) + events.push(ev); })(); // Wait until the first exec's sink is registered and its ServerFrame{exec} written, @@ -102,7 +153,8 @@ describe("relay Exec/Abort routing", () => { await expect( (async () => { - for await (const _ of relay.routeExec("sbx-1", 7, "echo hi", new Uint8Array(), 0, true)) void _; + for await (const _ of relay.routeExec('sbx-1', 7, 'echo hi', new Uint8Array(), 0, true)) + void _; })(), ).rejects.toThrow(/req_id 7 already in flight/); diff --git a/packages/session-backend/NOTES-pi-sessionmanager.md b/packages/session-backend/NOTES-pi-sessionmanager.md index 4adca3d..29ffc43 100644 --- a/packages/session-backend/NOTES-pi-sessionmanager.md +++ b/packages/session-backend/NOTES-pi-sessionmanager.md @@ -13,13 +13,13 @@ All line references target the pinned commit `406a2214` of `pi-fork/`. The class is the single owner of all JSONL file I/O for session data. It is instantiated via four static factory methods: -| Factory | Purpose | Line | -|---------|---------|------| -| `SessionManager.create(cwd, sessionDir?, options?)` | New session | 1385 | -| `SessionManager.open(path, sessionDir?, cwdOverride?)` | Open existing file | 1396 | -| `SessionManager.continueRecent(cwd, sessionDir?)` | Most-recent or new | 1412 | -| `SessionManager.inMemory(cwd?)` | No-persist (testing) | 1423 | -| `SessionManager.forkFrom(sourcePath, targetCwd, sessionDir?, options?)` | Fork a session | 1434 | +| Factory | Purpose | Line | +| ----------------------------------------------------------------------- | -------------------- | ---- | +| `SessionManager.create(cwd, sessionDir?, options?)` | New session | 1385 | +| `SessionManager.open(path, sessionDir?, cwdOverride?)` | Open existing file | 1396 | +| `SessionManager.continueRecent(cwd, sessionDir?)` | Most-recent or new | 1412 | +| `SessionManager.inMemory(cwd?)` | No-persist (testing) | 1423 | +| `SessionManager.forkFrom(sourcePath, targetCwd, sessionDir?, options?)` | Fork a session | 1434 | --- @@ -30,12 +30,14 @@ It is instantiated via four static factory methods: ``` private _appendEntry(entry: SessionEntry): void // line 937 ``` + Pushes to in-memory `fileEntries`, updates `byId` map, advances `leafId`, then calls `_persist(entry)`. ``` _persist(entry: SessionEntry): void // line 908 ``` + - If no assistant message has been written yet: holds in memory (lazy flush). - On first assistant message: opens file with flag `"wx"` and rewrites ALL accumulated entries at once (`writeFileSync`), then sets `flushed = true`. @@ -46,29 +48,30 @@ _persist(entry: SessionEntry): void // line 908 All return the new entry's `id: string`. -| Method | Signature | Line | -|--------|-----------|------| -| `appendMessage` | `(message: Message \| CustomMessage \| BashExecutionMessage): string` | 950 | -| `appendThinkingLevelChange` | `(thinkingLevel: string): string` | 963 | -| `appendModelChange` | `(provider: string, modelId: string): string` | 976 | -| `appendCompaction` | `(summary, firstKeptEntryId, tokensBefore, details?, fromHook?): string` | 990 | -| `appendCustomEntry` | `(customType: string, data?: unknown): string` | 1013 | -| `appendSessionInfo` | `(name: string): string` | 1027 | -| `appendCustomMessageEntry` | `(customType, content, display, details?): string` | 1061 | -| `appendLabelChange` | `(targetId, label: string \| undefined): string` | 1122 | -| `branchWithSummary` | `(branchFromId: string \| null, summary, details?, fromHook?): string` | 1262 | +| Method | Signature | Line | +| --------------------------- | ------------------------------------------------------------------------ | ---- | +| `appendMessage` | `(message: Message \| CustomMessage \| BashExecutionMessage): string` | 950 | +| `appendThinkingLevelChange` | `(thinkingLevel: string): string` | 963 | +| `appendModelChange` | `(provider: string, modelId: string): string` | 976 | +| `appendCompaction` | `(summary, firstKeptEntryId, tokensBefore, details?, fromHook?): string` | 990 | +| `appendCustomEntry` | `(customType: string, data?: unknown): string` | 1013 | +| `appendSessionInfo` | `(name: string): string` | 1027 | +| `appendCustomMessageEntry` | `(customType, content, display, details?): string` | 1061 | +| `appendLabelChange` | `(targetId, label: string \| undefined): string` | 1122 | +| `branchWithSummary` | `(branchFromId: string \| null, summary, details?, fromHook?): string` | 1262 | ### Compaction / rewrite path `_rewriteFile(): void` — line 872 — opens the file with flag `"w"` (truncate) and writes ALL `fileEntries` line-by-line. Called when: + - Migration is needed on load (`setSessionFile`, line 811). - A branched session is created with an assistant message already present (`createBranchedSession`, line 1350). **Compaction is NOT a rewrite.** The compaction flow calls `appendCompaction(summary, firstKeptEntryId, tokensBefore, ...)` which is a normal append. It records the summary text and the ID of the first entry to -keep; the *in-memory* tree then respects `firstKeptEntryId` when building +keep; the _in-memory_ tree then respects `firstKeptEntryId` when building context, but no entries are deleted from disk. The file remains append-only. Branching operations (`branch()`, `branchWithSummary()`) similarly only change the in-memory `leafId` pointer and optionally append a `branch_summary` @@ -78,35 +81,37 @@ entry — no truncation. ## 3. Read methods -| Method | Signature | Return shape | Line | -|--------|-----------|-------------|------| -| `getEntries()` | `(): SessionEntry[]` | All non-header entries, shallow copy | 1182 | -| `getEntry(id)` | `(id: string): SessionEntry \| undefined` | Single entry by id | 1093 | -| `getBranch(fromId?)` | `(fromId?: string): SessionEntry[]` | Path from root to leaf | 1150 | -| `getLeafEntry()` | `(): SessionEntry \| undefined` | Current leaf | 1089 | -| `getLeafId()` | `(): string \| null` | Current leaf id | 1085 | -| `getChildren(parentId)` | `(parentId: string): SessionEntry[]` | Direct children | 1100 | -| `getTree()` | `(): SessionTreeNode[]` | Full tree as nodes | 1191 | -| `getHeader()` | `(): SessionHeader \| null` | Session header record | 1172 | -| `buildSessionContext()` | `(): SessionContext` | LLM messages + model/thinking | 1165 | +| Method | Signature | Return shape | Line | +| ----------------------- | ----------------------------------------- | ------------------------------------ | ---- | +| `getEntries()` | `(): SessionEntry[]` | All non-header entries, shallow copy | 1182 | +| `getEntry(id)` | `(id: string): SessionEntry \| undefined` | Single entry by id | 1093 | +| `getBranch(fromId?)` | `(fromId?: string): SessionEntry[]` | Path from root to leaf | 1150 | +| `getLeafEntry()` | `(): SessionEntry \| undefined` | Current leaf | 1089 | +| `getLeafId()` | `(): string \| null` | Current leaf id | 1085 | +| `getChildren(parentId)` | `(parentId: string): SessionEntry[]` | Direct children | 1100 | +| `getTree()` | `(): SessionTreeNode[]` | Full tree as nodes | 1191 | +| `getHeader()` | `(): SessionHeader \| null` | Session header record | 1172 | +| `buildSessionContext()` | `(): SessionContext` | LLM messages + model/thinking | 1165 | `SessionEntry` union type — line 140: + ```typescript type SessionEntry = - | SessionMessageEntry // type: "message" - | ThinkingLevelChangeEntry // type: "thinking_level_change" - | ModelChangeEntry // type: "model_change" - | CompactionEntry // type: "compaction" - | BranchSummaryEntry // type: "branch_summary" - | CustomEntry // type: "custom" - | CustomMessageEntry // type: "custom_message" - | LabelEntry // type: "label" - | SessionInfoEntry // type: "session_info" + | SessionMessageEntry // type: "message" + | ThinkingLevelChangeEntry // type: "thinking_level_change" + | ModelChangeEntry // type: "model_change" + | CompactionEntry // type: "compaction" + | BranchSummaryEntry // type: "branch_summary" + | CustomEntry // type: "custom" + | CustomMessageEntry // type: "custom_message" + | LabelEntry // type: "label" + | SessionInfoEntry; // type: "session_info" ``` Every entry has `{ id: string; parentId: string | null; timestamp: string }` base fields — line 46. Static list helpers: + - `SessionManager.list(cwd, sessionDir?, onProgress?)` — line 1493 - `SessionManager.listAll(sessionDir?, onProgress?)` — line 1508 @@ -119,8 +124,8 @@ Static list helpers: ```typescript // line 438 function getDefaultSessionDirPath(cwd: string, agentDir: string): string { - const safePath = `--${resolvedCwd.replace(/^[/\\]/, "").replace(/[/\\:]/g, "-")}--`; - return join(agentDir, "sessions", safePath); + const safePath = `--${resolvedCwd.replace(/^[/\\]/, '').replace(/[/\\:]/g, '-')}--`; + return join(agentDir, 'sessions', safePath); } ``` @@ -133,7 +138,7 @@ So the default session dir for cwd `/home/user/project` is: ```typescript // line 844-846 (inside newSession()) -const fileTimestamp = timestamp.replace(/[:.]/g, "-"); +const fileTimestamp = timestamp.replace(/[:.]/g, '-'); this.sessionFile = join(this.getSessionDir(), `${fileTimestamp}_${this.sessionId}.jsonl`); ``` @@ -197,12 +202,16 @@ in `BashToolOptions` (bash.ts:134) that can rewrite command/cwd/env. ```typescript // bash.ts:40 interface BashOperations { - exec: (command: string, cwd: string, options: { - onData: (data: Buffer) => void; - signal?: AbortSignal; - timeout?: number; - env?: NodeJS.ProcessEnv; - }) => Promise<{ exitCode: number | null }>; + exec: ( + command: string, + cwd: string, + options: { + onData: (data: Buffer) => void; + signal?: AbortSignal; + timeout?: number; + env?: NodeJS.ProcessEnv; + }, + ) => Promise<{ exitCode: number | null }>; } // read.ts:43 @@ -228,7 +237,9 @@ interface WriteOperations { // ls.ts:32 interface LsOperations { exists: (absolutePath: string) => Promise | boolean; - stat: (absolutePath: string) => Promise<{ isDirectory: () => boolean }> | { isDirectory: () => boolean }; + stat: ( + absolutePath: string, + ) => Promise<{ isDirectory: () => boolean }> | { isDirectory: () => boolean }; readdir: (absolutePath: string) => Promise | string[]; } @@ -241,7 +252,11 @@ interface GrepOperations { // find.ts:41 interface FindOperations { exists: (absolutePath: string) => Promise | boolean; - glob: (pattern: string, cwd: string, options: { ignore: string[]; limit: number }) => Promise | string[]; + glob: ( + pattern: string, + cwd: string, + options: { ignore: string[]; limit: number }, + ) => Promise | string[]; } ``` @@ -254,36 +269,36 @@ These are registered via `runtime.on(eventName, handler)` on the extension ### Events confirmed in source -| Event name | Type interface | Blockable / cancellable | Location | -|---|---|---|---| -| `session_start` | `SessionStartEvent` | No | types.ts:1127 | -| `session_before_switch` | `SessionBeforeSwitchEvent` | Yes (result) | types.ts:1129 | -| `session_before_fork` | `SessionBeforeForkEvent` | Yes (result) | types.ts:1132 | -| `session_before_compact` | `SessionBeforeCompactEvent` | Yes (result) | types.ts:1134 | -| `session_compact` | `SessionCompactEvent` | No | types.ts:1137 | -| `session_shutdown` | `SessionShutdownEvent` | No | types.ts:1138 | -| `session_before_tree` | `SessionBeforeTreeEvent` | Yes (result) | types.ts:1139 | -| `session_tree` | `SessionTreeEvent` | No | types.ts:1140 | -| `context` | `ContextEvent` | Yes (result) | types.ts:1141 | -| `before_provider_request` | `BeforeProviderRequestEvent` | Yes (result) | types.ts:1143 | -| `after_provider_response` | `AfterProviderResponseEvent` | No | types.ts:1146 | -| `before_agent_start` | `BeforeAgentStartEvent` | Yes (result) | types.ts:1147 | -| `agent_start` | `AgentStartEvent` | No | types.ts:1148 | -| `agent_end` | `AgentEndEvent` | No | types.ts:1149 | -| `turn_start` | `TurnStartEvent` | No | types.ts:1150 | -| `turn_end` | `TurnEndEvent` | No | types.ts:1151 | -| `message_start` | `MessageStartEvent` | No | types.ts:1152 | -| `message_update` | `MessageUpdateEvent` | No | types.ts:1153 | -| `message_end` | `MessageEndEvent` | Yes (result) | types.ts:1154 | -| `tool_execution_start` | `ToolExecutionStartEvent` | No | types.ts:1155 | -| `tool_execution_update` | `ToolExecutionUpdateEvent` | No | types.ts:1156 | -| `tool_execution_end` | `ToolExecutionEndEvent` | No | types.ts:1157 | -| `model_select` | `ModelSelectEvent` | No | types.ts:1158 | -| `thinking_level_select` | `ThinkingLevelSelectEvent` | No | types.ts:1159 | -| `tool_call` | `ToolCallEvent` (discriminated union by toolName) | Yes — `input` mutable | types.ts:1160 | -| `tool_result` | `ToolResultEvent` (discriminated union by toolName) | Yes (result) | types.ts:1161 | -| `user_bash` | `UserBashEvent` | Yes (result) | types.ts:1162 | -| `input` | `InputEvent` | Yes (result) | types.ts:1163 | +| Event name | Type interface | Blockable / cancellable | Location | +| ------------------------- | --------------------------------------------------- | ----------------------- | ------------- | +| `session_start` | `SessionStartEvent` | No | types.ts:1127 | +| `session_before_switch` | `SessionBeforeSwitchEvent` | Yes (result) | types.ts:1129 | +| `session_before_fork` | `SessionBeforeForkEvent` | Yes (result) | types.ts:1132 | +| `session_before_compact` | `SessionBeforeCompactEvent` | Yes (result) | types.ts:1134 | +| `session_compact` | `SessionCompactEvent` | No | types.ts:1137 | +| `session_shutdown` | `SessionShutdownEvent` | No | types.ts:1138 | +| `session_before_tree` | `SessionBeforeTreeEvent` | Yes (result) | types.ts:1139 | +| `session_tree` | `SessionTreeEvent` | No | types.ts:1140 | +| `context` | `ContextEvent` | Yes (result) | types.ts:1141 | +| `before_provider_request` | `BeforeProviderRequestEvent` | Yes (result) | types.ts:1143 | +| `after_provider_response` | `AfterProviderResponseEvent` | No | types.ts:1146 | +| `before_agent_start` | `BeforeAgentStartEvent` | Yes (result) | types.ts:1147 | +| `agent_start` | `AgentStartEvent` | No | types.ts:1148 | +| `agent_end` | `AgentEndEvent` | No | types.ts:1149 | +| `turn_start` | `TurnStartEvent` | No | types.ts:1150 | +| `turn_end` | `TurnEndEvent` | No | types.ts:1151 | +| `message_start` | `MessageStartEvent` | No | types.ts:1152 | +| `message_update` | `MessageUpdateEvent` | No | types.ts:1153 | +| `message_end` | `MessageEndEvent` | Yes (result) | types.ts:1154 | +| `tool_execution_start` | `ToolExecutionStartEvent` | No | types.ts:1155 | +| `tool_execution_update` | `ToolExecutionUpdateEvent` | No | types.ts:1156 | +| `tool_execution_end` | `ToolExecutionEndEvent` | No | types.ts:1157 | +| `model_select` | `ModelSelectEvent` | No | types.ts:1158 | +| `thinking_level_select` | `ThinkingLevelSelectEvent` | No | types.ts:1159 | +| `tool_call` | `ToolCallEvent` (discriminated union by toolName) | Yes — `input` mutable | types.ts:1160 | +| `tool_result` | `ToolResultEvent` (discriminated union by toolName) | Yes (result) | types.ts:1161 | +| `user_bash` | `UserBashEvent` | Yes (result) | types.ts:1162 | +| `input` | `InputEvent` | Yes (result) | types.ts:1163 | **Note on `tool_call` blocking:** The handler receives a mutable `event.input` which can be patched in-place. The comment at types.ts:855 confirms: "Later @@ -306,6 +321,7 @@ pi --mode json # reads prompt from stdin when not a TTY ``` The `runPrintMode(runtimeHost, options)` function (print-mode.ts:32): + - Sends `initialMessage` via `session.prompt(...)` then any additional `messages`. - In `text` mode: prints last assistant message's text content to stdout. - In `json` mode: streams all `AgentSessionEvent` objects as JSON lines. @@ -377,27 +393,27 @@ type is unnecessary since Pi constructs the full entry before persistence. ## 10. Plan-assumption drift -| Plan identifier | Status | Real name / notes | -|---|---|---| -| `appendEntry` (on SessionManager) | DIFFERENT | Pi does NOT have a public `appendEntry` method. The public API is `appendMessage()`, `appendCompaction()`, `appendCustomEntry()`, etc. `_appendEntry` is private. The `appendEntry` that appears in extension types (types.ts:1230) is on `PiRuntime` (the extension API object) — it lets extensions append a custom entry to the session log. | -| `getEntries` (on SessionManager) | MATCH | `getEntries(): SessionEntry[]` — line 1182 | -| `ReadOperations` | MATCH | Interface exists, exported — read.ts:43 | -| `WriteOperations` | MATCH | Interface exists, exported — write.ts:25 | -| `EditOperations` | MATCH | Interface exists, exported — edit.ts:74 | -| `BashOperations` | MATCH | Interface exists, exported — bash.ts:40 | -| `LsOperations` | MATCH | Interface exists, exported — ls.ts:32 | -| `GrepOperations` | MATCH | Interface exists, exported — grep.ts:51 | -| `FindOperations` | MATCH | Interface exists, exported — find.ts:41 | -| Operations injection via `createBashTool(cwd, { operations })` | MATCH | Pattern confirmed for all seven tools | -| `tool_call` event (blockable) | MATCH | Confirmed — mutable `event.input` | -| `tool_result` event | MATCH | Confirmed | -| `turn_end` event | MATCH | Confirmed | -| `session_compact` event | MATCH | Confirmed | -| `session_before_compact` event | MATCH | Confirmed (blockable/cancellable) | -| `session_shutdown` event | MATCH | Confirmed | -| Headless flag `--print` / `-p` | MATCH | Confirmed in args.ts:14 and print-mode.ts | -| `--mode json` (NDJSON stream) | ADDITIONAL | Not in plan; exists as a second headless mode — event stream rather than final-text only. Highly useful for the HTTP turn handler. | -| `spawnHook` for operations injection | ADDITIONAL | BashToolOptions has `spawnHook?: BashSpawnHook` (bash.ts:134) — rewrites command/cwd/env before spawn. Alternative injection path for bash (no subprocess replacement needed, just command rewriting). | -| `session_start` event | ADDITIONAL | Not in plan list; fires on startup/reload/new/resume/fork — needed for cold-start reconstruction. | -| `before_agent_start` event | ADDITIONAL | Fires after user submits prompt but before agent loop — useful for budget voter. | -| `context` event | ADDITIONAL | Fires before each LLM call, can mutate messages — useful for injecting reconstructed context. | +| Plan identifier | Status | Real name / notes | +| -------------------------------------------------------------- | ---------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `appendEntry` (on SessionManager) | DIFFERENT | Pi does NOT have a public `appendEntry` method. The public API is `appendMessage()`, `appendCompaction()`, `appendCustomEntry()`, etc. `_appendEntry` is private. The `appendEntry` that appears in extension types (types.ts:1230) is on `PiRuntime` (the extension API object) — it lets extensions append a custom entry to the session log. | +| `getEntries` (on SessionManager) | MATCH | `getEntries(): SessionEntry[]` — line 1182 | +| `ReadOperations` | MATCH | Interface exists, exported — read.ts:43 | +| `WriteOperations` | MATCH | Interface exists, exported — write.ts:25 | +| `EditOperations` | MATCH | Interface exists, exported — edit.ts:74 | +| `BashOperations` | MATCH | Interface exists, exported — bash.ts:40 | +| `LsOperations` | MATCH | Interface exists, exported — ls.ts:32 | +| `GrepOperations` | MATCH | Interface exists, exported — grep.ts:51 | +| `FindOperations` | MATCH | Interface exists, exported — find.ts:41 | +| Operations injection via `createBashTool(cwd, { operations })` | MATCH | Pattern confirmed for all seven tools | +| `tool_call` event (blockable) | MATCH | Confirmed — mutable `event.input` | +| `tool_result` event | MATCH | Confirmed | +| `turn_end` event | MATCH | Confirmed | +| `session_compact` event | MATCH | Confirmed | +| `session_before_compact` event | MATCH | Confirmed (blockable/cancellable) | +| `session_shutdown` event | MATCH | Confirmed | +| Headless flag `--print` / `-p` | MATCH | Confirmed in args.ts:14 and print-mode.ts | +| `--mode json` (NDJSON stream) | ADDITIONAL | Not in plan; exists as a second headless mode — event stream rather than final-text only. Highly useful for the HTTP turn handler. | +| `spawnHook` for operations injection | ADDITIONAL | BashToolOptions has `spawnHook?: BashSpawnHook` (bash.ts:134) — rewrites command/cwd/env before spawn. Alternative injection path for bash (no subprocess replacement needed, just command rewriting). | +| `session_start` event | ADDITIONAL | Not in plan list; fires on startup/reload/new/resume/fork — needed for cold-start reconstruction. | +| `before_agent_start` event | ADDITIONAL | Fires after user submits prompt but before agent loop — useful for budget voter. | +| `context` event | ADDITIONAL | Fires before each LLM call, can mutate messages — useful for injecting reconstructed context. | diff --git a/packages/session-backend/src/backend.ts b/packages/session-backend/src/backend.ts index 9846028..d58acb0 100644 --- a/packages/session-backend/src/backend.ts +++ b/packages/session-backend/src/backend.ts @@ -1,5 +1,5 @@ // packages/session-backend/src/backend.ts -import type { StoredEntry } from "./entry"; +import type { StoredEntry } from './entry'; /** * Generic append-only log store. Entry-agnostic: stores opaque `E` records keyed diff --git a/packages/session-backend/src/entry.ts b/packages/session-backend/src/entry.ts index 8cf4c33..a322e1a 100644 --- a/packages/session-backend/src/entry.ts +++ b/packages/session-backend/src/entry.ts @@ -1,17 +1,17 @@ // packages/session-backend/src/entry.ts -import { createHash } from "node:crypto"; +import { createHash } from 'node:crypto'; /** * A stored log record: a thin envelope around an opaque harness-native entry. * The store never interprets `entry` except via the denormalized `piType`. */ export interface StoredEntry { - position: number; // monotonic 1-based offset; powers read(fromPosition) + position: number; // monotonic 1-based offset; powers read(fromPosition) session_id: string; - piType: string; // denormalized copy of the entry's discriminant, for cheap filtering - entry: E; // harness-native entry, stored & returned verbatim - content_sha256: string; // integrity hash of `entry` (canonical JSON) - timestamp: number; // wall-clock ms; audit only, not ordering + piType: string; // denormalized copy of the entry's discriminant, for cheap filtering + entry: E; // harness-native entry, stored & returned verbatim + content_sha256: string; // integrity hash of `entry` (canonical JSON) + timestamp: number; // wall-clock ms; audit only, not ordering } export function makeStoredEntry(args: { @@ -27,7 +27,7 @@ export function makeStoredEntry(args: { session_id: args.session_id, piType: args.piType, entry: args.entry, - content_sha256: createHash("sha256").update(payload).digest("hex"), + content_sha256: createHash('sha256').update(payload).digest('hex'), timestamp: args.timestamp ?? 0, // caller stamps real time; 0 keeps this pure/testable }; } diff --git a/packages/session-backend/src/index.ts b/packages/session-backend/src/index.ts index 9a25d00..b29180b 100644 --- a/packages/session-backend/src/index.ts +++ b/packages/session-backend/src/index.ts @@ -1,5 +1,5 @@ // packages/session-backend/src/index.ts -export type { StoredEntry } from "./entry"; -export { makeStoredEntry } from "./entry"; -export type { LogStore } from "./backend"; -export { RedisSessionBackend } from "./redis-backend"; +export type { StoredEntry } from './entry'; +export { makeStoredEntry } from './entry'; +export type { LogStore } from './backend'; +export { RedisSessionBackend } from './redis-backend'; diff --git a/packages/session-backend/src/redis-backend.ts b/packages/session-backend/src/redis-backend.ts index 457e429..ebf509a 100644 --- a/packages/session-backend/src/redis-backend.ts +++ b/packages/session-backend/src/redis-backend.ts @@ -1,7 +1,7 @@ // packages/session-backend/src/redis-backend.ts -import { createClient, type RedisClientType } from "redis"; -import { makeStoredEntry, type StoredEntry } from "./entry"; -import type { LogStore } from "./backend"; +import { createClient, type RedisClientType } from 'redis'; +import { makeStoredEntry, type StoredEntry } from './entry'; +import type { LogStore } from './backend'; const streamKey = (sid: string) => `session:${sid}`; const seqKey = (sid: string) => `session:${sid}:seq`; @@ -19,7 +19,7 @@ const seqKey = (sid: string) => `session:${sid}:seq`; export class RedisSessionBackend implements LogStore { private client: RedisClientType; private ready: Promise; - constructor(url = process.env.REDIS_URL ?? "redis://127.0.0.1:6379") { + constructor(url = process.env.REDIS_URL ?? 'redis://127.0.0.1:6379') { this.client = createClient({ url }); this.ready = this.client.connect().then(() => undefined); } @@ -32,7 +32,13 @@ export class RedisSessionBackend implements LogStore { async append(sid: string, entry: E, piType: string): Promise> { await this.ready; const position = await this.nextPosition(sid); - const stored = makeStoredEntry({ position, session_id: sid, piType, entry, timestamp: Date.now() }); + const stored = makeStoredEntry({ + position, + session_id: sid, + piType, + entry, + timestamp: Date.now(), + }); await this.client.xAdd(streamKey(sid), `${stored.position}-0`, { position: String(stored.position), timestamp: String(stored.timestamp), @@ -45,8 +51,8 @@ export class RedisSessionBackend implements LogStore { async read(sid: string, fromPosition = 1): Promise[]> { await this.ready; - const start = fromPosition <= 1 ? "-" : `${fromPosition}-0`; - const rows = await this.client.xRange(streamKey(sid), start, "+"); + const start = fromPosition <= 1 ? '-' : `${fromPosition}-0`; + const rows = await this.client.xRange(streamKey(sid), start, '+'); return rows.map((r): StoredEntry => ({ position: Number(r.message.position), timestamp: Number(r.message.timestamp), @@ -73,8 +79,8 @@ export class RedisSessionBackend implements LogStore { async list(): Promise { await this.ready; - const keys = await this.client.keys("session:*"); - return keys.filter((k) => !k.endsWith(":seq")).map((k) => k.slice("session:".length)); + const keys = await this.client.keys('session:*'); + return keys.filter((k) => !k.endsWith(':seq')).map((k) => k.slice('session:'.length)); } /** Test helper: delete a session's stream + sequence counter. */ diff --git a/packages/session-backend/test/entry.test.ts b/packages/session-backend/test/entry.test.ts index dd2b38f..ff951c8 100644 --- a/packages/session-backend/test/entry.test.ts +++ b/packages/session-backend/test/entry.test.ts @@ -1,29 +1,31 @@ -import { describe, it, expect } from "vitest"; -import { makeStoredEntry } from "../src/entry"; +import { describe, it, expect } from 'vitest'; +import { makeStoredEntry } from '../src/entry'; -describe("makeStoredEntry", () => { - it("wraps an opaque entry and hashes its JSON", () => { +describe('makeStoredEntry', () => { + it('wraps an opaque entry and hashes its JSON', () => { const e = makeStoredEntry({ - position: 1, session_id: "s", piType: "message", - entry: { type: "message", x: 1 }, + position: 1, + session_id: 's', + piType: 'message', + entry: { type: 'message', x: 1 }, }); expect(e.position).toBe(1); - expect(e.session_id).toBe("s"); - expect(e.piType).toBe("message"); - expect(e.entry).toEqual({ type: "message", x: 1 }); + expect(e.session_id).toBe('s'); + expect(e.piType).toBe('message'); + expect(e.entry).toEqual({ type: 'message', x: 1 }); expect(e.content_sha256).toMatch(/^[0-9a-f]{64}$/); expect(e.timestamp).toBe(0); // default when not provided }); - it("is deterministic: equal entries -> equal hash regardless of envelope fields", () => { - const a = makeStoredEntry({ position: 1, session_id: "s", piType: "t", entry: { a: 1 } }); - const b = makeStoredEntry({ position: 9, session_id: "s2", piType: "t", entry: { a: 1 } }); + it('is deterministic: equal entries -> equal hash regardless of envelope fields', () => { + const a = makeStoredEntry({ position: 1, session_id: 's', piType: 't', entry: { a: 1 } }); + const b = makeStoredEntry({ position: 9, session_id: 's2', piType: 't', entry: { a: 1 } }); expect(a.content_sha256).toBe(b.content_sha256); }); - it("different entry content -> different hash", () => { - const a = makeStoredEntry({ position: 1, session_id: "s", piType: "t", entry: { a: 1 } }); - const b = makeStoredEntry({ position: 1, session_id: "s", piType: "t", entry: { a: 2 } }); + it('different entry content -> different hash', () => { + const a = makeStoredEntry({ position: 1, session_id: 's', piType: 't', entry: { a: 1 } }); + const b = makeStoredEntry({ position: 1, session_id: 's', piType: 't', entry: { a: 2 } }); expect(a.content_sha256).not.toBe(b.content_sha256); }); }); diff --git a/packages/session-backend/test/redis-backend.test.ts b/packages/session-backend/test/redis-backend.test.ts index 53a7ae0..d13d52e 100644 --- a/packages/session-backend/test/redis-backend.test.ts +++ b/packages/session-backend/test/redis-backend.test.ts @@ -1,68 +1,76 @@ -import { describe, it, expect, beforeEach, afterAll } from "vitest"; -import { RedisSessionBackend } from "../src/redis-backend"; +import { describe, it, expect, beforeEach, afterAll } from 'vitest'; +import { RedisSessionBackend } from '../src/redis-backend'; -const SID = "test-" + process.pid; +const SID = 'test-' + process.pid; const b = new RedisSessionBackend<{ type: string; customType?: string; n?: number; id?: string }>( - "redis://127.0.0.1:6379", + 'redis://127.0.0.1:6379', ); -beforeEach(async () => { await b.reset(SID); }); -afterAll(async () => { await b.reset(SID); await b.close(); }); +beforeEach(async () => { + await b.reset(SID); +}); +afterAll(async () => { + await b.reset(SID); + await b.close(); +}); -describe("RedisSessionBackend (LogStore)", () => { - it("appends opaque entries and reads them back in order with monotonic positions", async () => { - await b.append(SID, { type: "message", n: 1 }, "message"); - await b.append(SID, { type: "message", n: 2 }, "message"); +describe('RedisSessionBackend (LogStore)', () => { + it('appends opaque entries and reads them back in order with monotonic positions', async () => { + await b.append(SID, { type: 'message', n: 1 }, 'message'); + await b.append(SID, { type: 'message', n: 2 }, 'message'); const rows = await b.read(SID); - expect(rows.map(r => r.position)).toEqual([1, 2]); - expect(rows.map(r => r.entry.n)).toEqual([1, 2]); - expect(rows[0].piType).toBe("message"); + expect(rows.map((r) => r.position)).toEqual([1, 2]); + expect(rows.map((r) => r.entry.n)).toEqual([1, 2]); + expect(rows[0].piType).toBe('message'); }); - it("read(fromPosition) returns only the tail", async () => { - await b.append(SID, { type: "message" }, "message"); - await b.append(SID, { type: "custom", customType: "checkpoint" }, "custom"); - await b.append(SID, { type: "message" }, "message"); + it('read(fromPosition) returns only the tail', async () => { + await b.append(SID, { type: 'message' }, 'message'); + await b.append(SID, { type: 'custom', customType: 'checkpoint' }, 'custom'); + await b.append(SID, { type: 'message' }, 'message'); const tail = await b.read(SID, 2); - expect(tail.map(r => r.position)).toEqual([2, 3]); + expect(tail.map((r) => r.position)).toEqual([2, 3]); }); - it("latestWhere returns the newest matching entry", async () => { - await b.append(SID, { type: "custom", customType: "checkpoint" }, "custom"); // pos 1 - await b.append(SID, { type: "message" }, "message"); // pos 2 - await b.append(SID, { type: "custom", customType: "checkpoint" }, "custom"); // pos 3 - const cp = await b.latestWhere(SID, e => e.type === "custom" && e.customType === "checkpoint"); + it('latestWhere returns the newest matching entry', async () => { + await b.append(SID, { type: 'custom', customType: 'checkpoint' }, 'custom'); // pos 1 + await b.append(SID, { type: 'message' }, 'message'); // pos 2 + await b.append(SID, { type: 'custom', customType: 'checkpoint' }, 'custom'); // pos 3 + const cp = await b.latestWhere( + SID, + (e) => e.type === 'custom' && e.customType === 'checkpoint', + ); expect(cp?.position).toBe(3); }); - it("returns the entry verbatim (round-trip identity)", async () => { - const entry = { type: "custom", customType: "checkpoint", n: 42 }; - await b.append(SID, entry, "custom"); + it('returns the entry verbatim (round-trip identity)', async () => { + const entry = { type: 'custom', customType: 'checkpoint', n: 42 }; + await b.append(SID, entry, 'custom'); const [row] = await b.read(SID); expect(row.entry).toEqual(entry); }); - it("read(fromPosition) uses a stream-id seek and returns exactly the tail", async () => { - await b.append(SID, { type: "message", n: 1 }, "message"); // pos 1 - await b.append(SID, { type: "message", n: 2 }, "message"); // pos 2 - await b.append(SID, { type: "message", n: 3 }, "message"); // pos 3 + it('read(fromPosition) uses a stream-id seek and returns exactly the tail', async () => { + await b.append(SID, { type: 'message', n: 1 }, 'message'); // pos 1 + await b.append(SID, { type: 'message', n: 2 }, 'message'); // pos 2 + await b.append(SID, { type: 'message', n: 3 }, 'message'); // pos 3 const tail = await b.read(SID, 2); expect(tail.map((r) => r.position)).toEqual([2, 3]); expect(tail.map((r) => r.entry.n)).toEqual([2, 3]); }); - it("read(fromPosition) past the end returns empty", async () => { - await b.append(SID, { type: "message", n: 1 }, "message"); // pos 1 + it('read(fromPosition) past the end returns empty', async () => { + await b.append(SID, { type: 'message', n: 1 }, 'message'); // pos 1 const tail = await b.read(SID, 99); expect(tail).toEqual([]); }); - it("positionOfId returns the position of the entry with the matching id, or null", async () => { - await b.append(SID, { type: "message", id: "a" }, "message"); // pos 1 - await b.append(SID, { type: "message", id: "b" }, "message"); // pos 2 - await b.append(SID, { type: "compaction", id: "c" }, "compaction"); // pos 3 - expect(await b.positionOfId(SID, "b")).toBe(2); - expect(await b.positionOfId(SID, "c")).toBe(3); - expect(await b.positionOfId(SID, "missing")).toBeNull(); + it('positionOfId returns the position of the entry with the matching id, or null', async () => { + await b.append(SID, { type: 'message', id: 'a' }, 'message'); // pos 1 + await b.append(SID, { type: 'message', id: 'b' }, 'message'); // pos 2 + await b.append(SID, { type: 'compaction', id: 'c' }, 'compaction'); // pos 3 + expect(await b.positionOfId(SID, 'b')).toBe(2); + expect(await b.positionOfId(SID, 'c')).toBe(3); + expect(await b.positionOfId(SID, 'missing')).toBeNull(); }); }); diff --git a/packages/work-queue/package.json b/packages/work-queue/package.json index 74f2014..565271d 100644 --- a/packages/work-queue/package.json +++ b/packages/work-queue/package.json @@ -2,8 +2,17 @@ "name": "@sh/work-queue", "type": "module", "version": "0.0.0", - "exports": { ".": "./src/index.ts" }, - "scripts": { "test": "vitest run" }, - "devDependencies": { "typescript": "^5.5.0", "vitest": "^2.0.0" }, - "dependencies": { "redis": "^6.0.0" } + "exports": { + ".": "./src/index.ts" + }, + "scripts": { + "test": "vitest run" + }, + "devDependencies": { + "typescript": "^5.5.0", + "vitest": "^2.0.0" + }, + "dependencies": { + "redis": "^6.0.0" + } } diff --git a/packages/work-queue/src/index.ts b/packages/work-queue/src/index.ts index b632667..b7e060c 100644 --- a/packages/work-queue/src/index.ts +++ b/packages/work-queue/src/index.ts @@ -1,2 +1,2 @@ -export { RedisWorkQueue } from "./queue.js"; -export type { WorkQueue, ClaimedEntry } from "./queue.js"; +export { RedisWorkQueue } from './queue.js'; +export type { WorkQueue, ClaimedEntry } from './queue.js'; diff --git a/packages/work-queue/src/queue.ts b/packages/work-queue/src/queue.ts index 8394be3..83cd33a 100644 --- a/packages/work-queue/src/queue.ts +++ b/packages/work-queue/src/queue.ts @@ -1,4 +1,4 @@ -import { createClient, type RedisClientType } from "redis"; +import { createClient, type RedisClientType } from 'redis'; export interface ClaimedEntry { entryId: string; @@ -9,13 +9,19 @@ export interface ClaimedEntry { export interface WorkQueue { ensureGroup(): Promise; enqueue(envelope: unknown): Promise; - claim(consumerId: string, opts: { minIdleMs: number; blockMs: number }): Promise; + claim( + consumerId: string, + opts: { minIdleMs: number; blockMs: number }, + ): Promise; ack(entryId: string): Promise; touch(entryId: string, consumerId: string): Promise; pending(): Promise; deleteConsumer(consumerId: string): Promise; gcIdleConsumers(minIdleMs: number): Promise; - reapDeadLetters(consumerId: string, opts: { minIdleMs: number; maxAttempts: number }): Promise>; + reapDeadLetters( + consumerId: string, + opts: { minIdleMs: number; maxAttempts: number }, + ): Promise>; purge(): Promise; close(): Promise; } @@ -24,9 +30,9 @@ export class RedisWorkQueue implements WorkQueue { private client: RedisClientType; private ready: Promise; constructor( - url = process.env.REDIS_URL ?? "redis://127.0.0.1:6379", - private readonly stream = "leaf-queue", - private readonly group = "leaf-workers", + url = process.env.REDIS_URL ?? 'redis://127.0.0.1:6379', + private readonly stream = 'leaf-queue', + private readonly group = 'leaf-workers', ) { this.client = createClient({ url }) as RedisClientType; this.ready = this.client.connect().then(() => undefined); @@ -36,27 +42,46 @@ export class RedisWorkQueue implements WorkQueue { await this.ready; try { // "0" = deliver from the start of the stream; MKSTREAM creates it if absent. - await this.client.xGroupCreate(this.stream, this.group, "0", { MKSTREAM: true }); + await this.client.xGroupCreate(this.stream, this.group, '0', { MKSTREAM: true }); } catch (err) { - if (!String((err as Error).message).includes("BUSYGROUP")) throw err; + if (!String((err as Error).message).includes('BUSYGROUP')) throw err; } } async enqueue(envelope: unknown): Promise { await this.ready; - return this.client.xAdd(this.stream, "*", { envelope: JSON.stringify(envelope) }); + return this.client.xAdd(this.stream, '*', { envelope: JSON.stringify(envelope) }); } - async claim(consumerId: string, opts: { minIdleMs: number; blockMs: number }): Promise { + async claim( + consumerId: string, + opts: { minIdleMs: number; blockMs: number }, + ): Promise { await this.ready; // 1. Prefer reclaiming a stale (delivered-but-unacked) entry — crash recovery. - const auto = await this.client.xAutoClaim(this.stream, this.group, consumerId, opts.minIdleMs, "0", { COUNT: 1 }); + const auto = await this.client.xAutoClaim( + this.stream, + this.group, + consumerId, + opts.minIdleMs, + '0', + { COUNT: 1 }, + ); const reclaimed = auto.messages?.find((m) => m && m.message); if (reclaimed) { - return { entryId: reclaimed.id, envelope: JSON.parse(reclaimed.message.envelope), deliveryCount: await this.deliveryCount(reclaimed.id) }; + return { + entryId: reclaimed.id, + envelope: JSON.parse(reclaimed.message.envelope), + deliveryCount: await this.deliveryCount(reclaimed.id), + }; } // 2. Otherwise read a brand-new entry. - const res = await this.client.xReadGroup(this.group, consumerId, [{ key: this.stream, id: ">" }], { COUNT: 1, BLOCK: opts.blockMs }); + const res = await this.client.xReadGroup( + this.group, + consumerId, + [{ key: this.stream, id: '>' }], + { COUNT: 1, BLOCK: opts.blockMs }, + ); const msg = res?.[0]?.messages?.[0]; if (!msg) return null; return { entryId: msg.id, envelope: JSON.parse(msg.message.envelope), deliveryCount: 1 }; @@ -105,19 +130,39 @@ export class RedisWorkQueue implements WorkQueue { // Bounded per startup; backlogs > REAP_BATCH drain across successive pod restarts. private static readonly REAP_BATCH = 100; - async reapDeadLetters(consumerId: string, opts: { minIdleMs: number; maxAttempts: number }): Promise> { + async reapDeadLetters( + consumerId: string, + opts: { minIdleMs: number; maxAttempts: number }, + ): Promise> { await this.ready; const deadLettered: Array<{ entryId: string; envelope: unknown }> = []; - const pending = await this.client.xPendingRange(this.stream, this.group, "-", "+", RedisWorkQueue.REAP_BATCH); + const pending = await this.client.xPendingRange( + this.stream, + this.group, + '-', + '+', + RedisWorkQueue.REAP_BATCH, + ); for (const entry of pending) { - if (entry.millisecondsSinceLastDelivery >= opts.minIdleMs && entry.deliveriesCounter > opts.maxAttempts) { - const claimed = await this.client.xClaim(this.stream, this.group, consumerId, opts.minIdleMs, [entry.id]); + if ( + entry.millisecondsSinceLastDelivery >= opts.minIdleMs && + entry.deliveriesCounter > opts.maxAttempts + ) { + const claimed = await this.client.xClaim( + this.stream, + this.group, + consumerId, + opts.minIdleMs, + [entry.id], + ); const msg = claimed?.[0]; if (!msg) continue; // entry was reclaimed by another consumer between inspect and claim let envelope: unknown = null; try { envelope = msg.message?.envelope ? JSON.parse(msg.message.envelope) : null; - } catch { /* malformed — still dead-letter it */ } + } catch { + /* malformed — still dead-letter it */ + } await this.client.xAck(this.stream, this.group, entry.id); deadLettered.push({ entryId: entry.id, envelope }); } @@ -127,8 +172,16 @@ export class RedisWorkQueue implements WorkQueue { async purge(): Promise { await this.ready; - try { await this.client.xGroupDestroy(this.stream, this.group); } catch { /* ignore */ } - try { await this.client.del(this.stream); } catch { /* ignore */ } + try { + await this.client.xGroupDestroy(this.stream, this.group); + } catch { + /* ignore */ + } + try { + await this.client.del(this.stream); + } catch { + /* ignore */ + } } async close(): Promise { diff --git a/packages/work-queue/test/queue.test.ts b/packages/work-queue/test/queue.test.ts index 37bcc26..48dd85d 100644 --- a/packages/work-queue/test/queue.test.ts +++ b/packages/work-queue/test/queue.test.ts @@ -1,69 +1,74 @@ -import { describe, it, expect, afterEach } from "vitest"; -import { RedisWorkQueue } from "../src/queue"; +import { describe, it, expect, afterEach } from 'vitest'; +import { RedisWorkQueue } from '../src/queue'; -const URL = process.env.REDIS_URL ?? "redis://127.0.0.1:6379"; +const URL = process.env.REDIS_URL ?? 'redis://127.0.0.1:6379'; let q: RedisWorkQueue; // unique stream per test so runs don't collide const stream = () => `test-leaf-queue-${process.pid}-${Math.floor(performance.now())}`; -afterEach(async () => { if (q) { await q.purge(); await q.close(); } }); +afterEach(async () => { + if (q) { + await q.purge(); + await q.close(); + } +}); -describe("RedisWorkQueue", () => { - it("enqueues and claims an entry with the envelope and deliveryCount 1", async () => { +describe('RedisWorkQueue', () => { + it('enqueues and claims an entry with the envelope and deliveryCount 1', async () => { q = new RedisWorkQueue(URL, stream()); await q.ensureGroup(); - await q.enqueue({ sessionId: "s1", inputsRef: "/in", resultRef: "/out" }); - const c = await q.claim("worker-a", { minIdleMs: 5000, blockMs: 200 }); + await q.enqueue({ sessionId: 's1', inputsRef: '/in', resultRef: '/out' }); + const c = await q.claim('worker-a', { minIdleMs: 5000, blockMs: 200 }); expect(c).not.toBeNull(); - expect((c!.envelope as any).sessionId).toBe("s1"); + expect((c!.envelope as any).sessionId).toBe('s1'); expect(c!.deliveryCount).toBe(1); }); - it("ack removes the entry from the pending list", async () => { + it('ack removes the entry from the pending list', async () => { q = new RedisWorkQueue(URL, stream()); await q.ensureGroup(); - await q.enqueue({ sessionId: "s2" }); - const c = await q.claim("worker-a", { minIdleMs: 5000, blockMs: 200 }); + await q.enqueue({ sessionId: 's2' }); + const c = await q.claim('worker-a', { minIdleMs: 5000, blockMs: 200 }); await q.ack(c!.entryId); expect(await q.pending()).toBe(0); }); - it("claim returns null when there is no new or reclaimable work", async () => { + it('claim returns null when there is no new or reclaimable work', async () => { q = new RedisWorkQueue(URL, stream()); await q.ensureGroup(); - const c = await q.claim("worker-a", { minIdleMs: 5000, blockMs: 100 }); + const c = await q.claim('worker-a', { minIdleMs: 5000, blockMs: 100 }); expect(c).toBeNull(); }); - it("reclaims an unacked entry for another consumer and bumps deliveryCount", async () => { + it('reclaims an unacked entry for another consumer and bumps deliveryCount', async () => { q = new RedisWorkQueue(URL, stream()); await q.ensureGroup(); - await q.enqueue({ sessionId: "s3" }); - const first = await q.claim("worker-a", { minIdleMs: 0, blockMs: 200 }); + await q.enqueue({ sessionId: 's3' }); + const first = await q.claim('worker-a', { minIdleMs: 0, blockMs: 200 }); expect(first!.deliveryCount).toBe(1); // worker-a never acks; worker-b reclaims with minIdle 0 - const second = await q.claim("worker-b", { minIdleMs: 0, blockMs: 200 }); + const second = await q.claim('worker-b', { minIdleMs: 0, blockMs: 200 }); expect(second!.entryId).toBe(first!.entryId); expect(second!.deliveryCount).toBe(2); }); - it("deleteConsumer removes a consumer from the group", async () => { + it('deleteConsumer removes a consumer from the group', async () => { q = new RedisWorkQueue(URL, stream()); await q.ensureGroup(); - await q.enqueue({ sessionId: "s4" }); - const c = await q.claim("worker-del", { minIdleMs: 5000, blockMs: 200 }); + await q.enqueue({ sessionId: 's4' }); + const c = await q.claim('worker-del', { minIdleMs: 5000, blockMs: 200 }); await q.ack(c!.entryId); - await q.deleteConsumer("worker-del"); + await q.deleteConsumer('worker-del'); // Consumer no longer exists — gcIdleConsumers should find nothing for it const removed = await q.gcIdleConsumers(0); expect(removed).toBe(0); }); - it("gcIdleConsumers removes consumers with 0 pending and idle >= threshold", async () => { + it('gcIdleConsumers removes consumers with 0 pending and idle >= threshold', async () => { q = new RedisWorkQueue(URL, stream()); await q.ensureGroup(); - await q.enqueue({ sessionId: "s5" }); - const c = await q.claim("worker-gc", { minIdleMs: 5000, blockMs: 200 }); + await q.enqueue({ sessionId: 's5' }); + const c = await q.claim('worker-gc', { minIdleMs: 5000, blockMs: 200 }); await q.ack(c!.entryId); // worker-gc now has 0 pending; after a tiny wait it's idle await new Promise((r) => setTimeout(r, 10)); @@ -71,40 +76,40 @@ describe("RedisWorkQueue", () => { expect(removed).toBe(1); }); - it("gcIdleConsumers does NOT remove consumers that still have pending entries", async () => { + it('gcIdleConsumers does NOT remove consumers that still have pending entries', async () => { q = new RedisWorkQueue(URL, stream()); await q.ensureGroup(); - await q.enqueue({ sessionId: "s6" }); - await q.claim("worker-busy", { minIdleMs: 5000, blockMs: 200 }); + await q.enqueue({ sessionId: 's6' }); + await q.claim('worker-busy', { minIdleMs: 5000, blockMs: 200 }); // worker-busy has 1 pending (unacked) const removed = await q.gcIdleConsumers(0); expect(removed).toBe(0); }); - it("reapDeadLetters ACKs entries past maxAttempts and returns their envelopes", async () => { + it('reapDeadLetters ACKs entries past maxAttempts and returns their envelopes', async () => { q = new RedisWorkQueue(URL, stream()); await q.ensureGroup(); - await q.enqueue({ sessionId: "s7", resultRef: "/out" }); + await q.enqueue({ sessionId: 's7', resultRef: '/out' }); // Simulate 4 deliveries by claiming+not-acking repeatedly (minIdle 0 to reclaim immediately) - await q.claim("w1", { minIdleMs: 0, blockMs: 200 }); - await q.claim("w2", { minIdleMs: 0, blockMs: 200 }); - await q.claim("w3", { minIdleMs: 0, blockMs: 200 }); - await q.claim("w4", { minIdleMs: 0, blockMs: 200 }); + await q.claim('w1', { minIdleMs: 0, blockMs: 200 }); + await q.claim('w2', { minIdleMs: 0, blockMs: 200 }); + await q.claim('w3', { minIdleMs: 0, blockMs: 200 }); + await q.claim('w4', { minIdleMs: 0, blockMs: 200 }); // Entry now has deliveryCount=4, idle resets on each claim but we use minIdleMs=0 expect(await q.pending()).toBe(1); - const dead = await q.reapDeadLetters("reaper", { minIdleMs: 0, maxAttempts: 3 }); + const dead = await q.reapDeadLetters('reaper', { minIdleMs: 0, maxAttempts: 3 }); expect(dead).toHaveLength(1); - expect((dead[0].envelope as any).sessionId).toBe("s7"); + expect((dead[0].envelope as any).sessionId).toBe('s7'); expect(await q.pending()).toBe(0); }); - it("reapDeadLetters leaves entries with deliveryCount <= maxAttempts", async () => { + it('reapDeadLetters leaves entries with deliveryCount <= maxAttempts', async () => { q = new RedisWorkQueue(URL, stream()); await q.ensureGroup(); - await q.enqueue({ sessionId: "s8" }); + await q.enqueue({ sessionId: 's8' }); // Only 1 delivery - await q.claim("w1", { minIdleMs: 5000, blockMs: 200 }); - const dead = await q.reapDeadLetters("reaper", { minIdleMs: 0, maxAttempts: 3 }); + await q.claim('w1', { minIdleMs: 5000, blockMs: 200 }); + const dead = await q.reapDeadLetters('reaper', { minIdleMs: 0, maxAttempts: 3 }); expect(dead).toHaveLength(0); expect(await q.pending()).toBe(1); }); diff --git a/pnpm-workspace.yaml b/pnpm-workspace.yaml index a6917c8..d991c25 100644 --- a/pnpm-workspace.yaml +++ b/pnpm-workspace.yaml @@ -1,4 +1,4 @@ packages: - - "packages/*" - - "harness" - - "experiments" + - 'packages/*' + - 'harness' + - 'experiments' diff --git a/remote-worker/DESIGN.md b/remote-worker/DESIGN.md index 59782f2..9f5a228 100644 --- a/remote-worker/DESIGN.md +++ b/remote-worker/DESIGN.md @@ -34,17 +34,17 @@ The worker **dials out** to the relay and keeps ONE full-duplex gRPC stream open ### Wire contract the worker must honor (proto §8) -| Rule | This worker | -|------|-------------| -| `Hello` first, with `sandbox_id` | ✅ before anything else; capabilities probed from PATH | -| stdout → `Chunk{STREAM_STDOUT}`, stderr → `STREAM_STDERR` | ✅ separate pipes, separate frames | -| `Chunk` capped so one frame stays small | ✅ 32 KiB, which is also the pipe read size | -| terminate each exec with `End{req_id, exit_code}` | ✅ real child exit code; `-1` when signalled | -| failures → `ExecError{req_id, message}` | ✅ spawn failures, and `timeout:` on expiry | -| `Abort` → SIGKILL the in-flight child | ✅ kills the whole process group (`Setpgid`) | -| worker-side `timeout_s` | ✅ SIGKILL at expiry → `ExecError{"timeout:"}` | +| Rule | This worker | +| ---------------------------------------------------------------------------------------- | ------------------------------------------------------------ | +| `Hello` first, with `sandbox_id` | ✅ before anything else; capabilities probed from PATH | +| stdout → `Chunk{STREAM_STDOUT}`, stderr → `STREAM_STDERR` | ✅ separate pipes, separate frames | +| `Chunk` capped so one frame stays small | ✅ 32 KiB, which is also the pipe read size | +| terminate each exec with `End{req_id, exit_code}` | ✅ real child exit code; `-1` when signalled | +| failures → `ExecError{req_id, message}` | ✅ spawn failures, and `timeout:` on expiry | +| `Abort` → SIGKILL the in-flight child | ✅ kills the whole process group (`Setpgid`) | +| worker-side `timeout_s` | ✅ SIGKILL at expiry → `ExecError{"timeout:"}` | | dedup / at-least-once: cache `req_id →` terminal frame (`End`, or a timeout `ExecError`) | ✅ bounded LRU (256), guarded by a command+stdin fingerprint | -| `Heartbeat` for liveness | ✅ every 15s | +| `Heartbeat` for liveness | ✅ every 15s | ## What it does on each Exec @@ -61,7 +61,7 @@ The worker **dials out** to the relay and keeps ONE full-duplex gRPC stream open 5. Streams stdout and stderr back as 32 KiB `Chunk` frames tagged with their stream. With `streaming: false` it buffers output and emits it at exit in the same 32 KiB-capped `Chunk` frames — one burst rather than incremental delivery. The - guarantee is *when* output is sent, not that it is a single frame: the cap + guarantee is _when_ output is sent, not that it is a single frame: the cap still applies, since an 8 MiB frame would exceed gRPC's default receive limit. 6. Terminates with `End{exit_code}`, or `ExecError{"timeout:"}` if `timeout_s` expired, or `End{-1}` if aborted. @@ -69,13 +69,13 @@ The worker **dials out** to the relay and keeps ONE full-duplex gRPC stream open There is no persistent shell: every command the harness sends is self-contained (`cd 'cwd' && …`), and a shared shell could not give each exec its own stdin EOF. -## Running locally on this laptop, against ykt1 ← the interesting part +## Running locally on this laptop, against ykt1 ← the interesting part The worker **dials the relay**; the relay never dials the worker. So a laptop worker does **not** need any inbound route — we just need the laptop to reach the relay. In-cluster that's the ClusterIP `sandbox-relay.default.svc:8443`; from a laptop we tunnel to it with `oc port-forward`. The harness→relay→worker execs -then ride *back down* the worker-initiated stream through the same tunnel. +then ride _back down_ the worker-initiated stream through the same tunnel. ``` laptop: remote-worker ──dial──▶ localhost:8443 ─┐ @@ -152,6 +152,7 @@ directly with the `grpcurl` exec above; the worker pod log shows the matching > exec is silently default-denied (harness→relay blocked) and times out. This repo's > `deploy/knative/harness-egress-policy.yaml` now adds the missing rule > (`app=sandbox-relay` :8443). If you deployed before that fix, patch it live: +> > ```bash > oc patch networkpolicy serverless-harness-egress -n default --type=json \ > -p '[{"op":"add","path":"/spec/egress/-","value":{"ports":[{"port":8443,"protocol":"TCP"}],"to":[{"podSelector":{"matchLabels":{"app":"sandbox-relay"}}}]}}]' @@ -162,7 +163,7 @@ directly with the `grpcurl` exec above; the worker pod log shows the matching harness-driven leaf, since the block is on the harness→relay leg regardless of where the worker runs. -For a worker on *other* infrastructure (not this cluster), expose the relay via an +For a worker on _other_ infrastructure (not this cluster), expose the relay via an OpenShift Route on :443 with TLS + HTTP/2 and point `RELAY_ADDR` at the Route host; the worker then dials with TLS (`RELAY_TLS=1`) instead of `insecure`. No code or binary changes are needed — the same worker image and the env vars in @@ -171,15 +172,15 @@ outside-the-cluster (`RELAY_TLS=1`, Route host) cases. ## Environment variables -| Var | Default | Meaning | -|-----|---------|---------| -| `RELAY_ADDR` | `localhost:8443` | relay address (tunnel, ClusterIP, or Route host) | -| `SANDBOX_ID` | `sbx-laptop-1` | stable id; one live Attach per id | -| `SANDBOX_TOKEN` | `dev-token` | Bearer token; must match the relay | -| `RELAY_TLS` | `0` | `1`/`true` to dial with TLS (for a Route :443); `0`/`false` = plaintext h2c. Anything else is **fatal** — it gates whether the bearer token crosses the wire in cleartext, so the worker refuses to guess | -| `WORKER_MAX_CONCURRENT` | `4` | dispatch pool size; also advertised as `Hello.capacity_max` | -| `SANDBOX_IMAGE` | (empty) | advertised in `Hello`; informational, not enforced by the worker | -| `SANDBOX_TRUST` | `untrusted` | advertised in `Hello`; informational, not enforced by the worker | +| Var | Default | Meaning | +| ----------------------- | ---------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | +| `RELAY_ADDR` | `localhost:8443` | relay address (tunnel, ClusterIP, or Route host) | +| `SANDBOX_ID` | `sbx-laptop-1` | stable id; one live Attach per id | +| `SANDBOX_TOKEN` | `dev-token` | Bearer token; must match the relay | +| `RELAY_TLS` | `0` | `1`/`true` to dial with TLS (for a Route :443); `0`/`false` = plaintext h2c. Anything else is **fatal** — it gates whether the bearer token crosses the wire in cleartext, so the worker refuses to guess | +| `WORKER_MAX_CONCURRENT` | `4` | dispatch pool size; also advertised as `Hello.capacity_max` | +| `SANDBOX_IMAGE` | (empty) | advertised in `Hello`; informational, not enforced by the worker | +| `SANDBOX_TRUST` | `untrusted` | advertised in `Hello`; informational, not enforced by the worker | ## Files diff --git a/remote-worker/worker-deployment.yaml b/remote-worker/worker-deployment.yaml index b118a44..97e6fe4 100644 --- a/remote-worker/worker-deployment.yaml +++ b/remote-worker/worker-deployment.yaml @@ -32,10 +32,10 @@ spec: value: __TOKEN__ securityContext: runAsNonRoot: true - runAsUser: 1001 # matches the image's USER 1001; nonroot-v2 SCC allows it + runAsUser: 1001 # matches the image's USER 1001; nonroot-v2 SCC allows it allowPrivilegeEscalation: false capabilities: - drop: ["ALL"] + drop: ['ALL'] seccompProfile: type: RuntimeDefault # Memory must cover the worker's own buffering, not just its idle size. @@ -47,5 +47,5 @@ spec: # A 64Mi limit was therefore an OOMKill the relay could trigger at will. # Keep this in sync with BufferCap in internal/exec/runner.go. resources: - requests: { cpu: "10m", memory: "64Mi" } - limits: { memory: "256Mi" } + requests: { cpu: '10m', memory: '64Mi' } + limits: { memory: '256Mi' } From 0409617511f19015f9d585e666ffdc2dcc45462c Mon Sep 17 00:00:00 2001 From: Paolo Dettori Date: Wed, 2 Sep 2026 10:36:12 -0400 Subject: [PATCH 3/3] fix(pre-commit): drop the prettier hook's `exclude`, keep one ignore list Review follow-up. `exclude` duplicated a subset of `.prettierignore` (`pi-fork/`, `packages/k8s-sandbox/src/gen/`, `gen/`) and omitted the rest (`node_modules/`, `dist/`, `*.log`, `pnpm-lock.yaml`) -- a second list that had to stay in agreement with the first. Prettier applies `.prettierignore` even to paths passed explicitly on the command line, which is exactly how pre-commit invokes it, so `.prettierignore` alone gives the same result. Verified rather than assumed: appending deliberately misformatted code to a file under `packages/k8s-sandbox/src/gen/` and passing that path directly to `pnpm exec prettier --write --ignore-unknown` leaves it untouched and exits 0 -- including when every path passed is ignored. This extends the property the pinned binary already gives (`make fmt` and the hook cannot disagree about formatting) to the ignore set: they now cannot disagree about what to skip either. `pre-commit run --all-files` still exits 0 with all nine hooks running, and no generated or submodule file is touched. Assisted-By: Claude (Anthropic AI) Signed-off-by: Paolo Dettori --- .pre-commit-config.yaml | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/.pre-commit-config.yaml b/.pre-commit-config.yaml index 686041e..200d05e 100644 --- a/.pre-commit-config.yaml +++ b/.pre-commit-config.yaml @@ -19,6 +19,11 @@ repos: # # `ts`/`tsx` are the identify tags for TypeScript -- there is no `typescript` tag, and # naming one silently matched no .ts file and then failed config validation outright. + # + # Deliberately no `exclude:`. Prettier applies `.prettierignore` even to paths handed to + # it explicitly, which is exactly how pre-commit invokes it, so `.prettierignore` is the + # single source of truth for the ignore set -- one list to keep correct instead of two + # that must agree. Same reasoning as the single pinned binary above. - repo: local hooks: - id: prettier @@ -26,7 +31,6 @@ repos: entry: pnpm exec prettier --write --ignore-unknown language: system types_or: [javascript, jsx, ts, tsx, json, yaml, markdown] - exclude: ^(pi-fork/|packages/k8s-sandbox/src/gen/|gen/) # `-S warning` matches the shellcheck gate in .github/workflows/security-scans.yml # exactly, so the hook and that job cannot disagree about what fails. The scripts are