Harness: memory, hooks, file-defined commands, and a /loop that lives in core - #48
Merged
Conversation
…r from reading as a missing field A `file_write` whose arguments the output cap cut in half came back as "missing required string path" — the model was sent to fix an argument it had spelled correctly. Three causes, all of them silent: - Unparseable argument text was thrown away (`unwrap_or(Value::Null)`) and reached the tool as an empty object. It is kept now as `ToolCall.unparsed_arguments` and refused before dispatch, with a message that tells a cut-off call apart from malformed JSON and names the cap. - `model.max_tokens` defaulted to 4096. One real source file is bigger than that, and on a reasoning model the thinking comes out of the same budget. Raised to 32768; a model whose own ceiling is lower now says so in its refusal and gets it (`rejected_output_cap`), so the raise cannot lock a smaller model out. - A provider that could not be reached said "model request failed for X" and dropped the cause. It now names the endpoint and the transport error, and for codex-router — where an OAuth user has no API key to fall back on — a connect refusal searches loopback for the gateway it moved to (4100 -> 4200 in 0.4.0-beta) and a 401 names the key file that rotated. Driven live on both provider paths: 26KB and 11KB single-call writes that the old cap could not have finished, the truncation refusal firing verbatim at a small cap, and a turn completing after core found the moved router itself. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Cru2ftKTALf6s62omapuw8
Two halves that turned out to be different problems. `spawn_ledger.rs` records what core spawned, and `computer.rs` refuses to drive anything else. Entries carry the kernel start time as well as the pid, because pids are recycled: a bare-pid ledger keeps vouching for a dead browser after the kernel reissues its number, and hands the agent whatever now owns it. On top of that boundary: `computer_targets`, `computer_doctor`, `computer_look`, `computer_type`, `computer_key`. `computer_click` and `computer_scroll` are built, measured, and withheld. Synthetic mouse events do not reach a background window by any route tried — CGEventPostToPid, CGEventPostToPSN, SkyLight's SLEventPostToPid, an SLPSPostEventRecordTo activation record, a primer click — while keyboard arrives through all of them, verified against a plain AppKit window that is never key. Shipping input that silently does nothing is worse than shipping none, so both stay under #[cfg(test)] with their failing live tests as the regression check. The current target is three.js and WebGPU, which live in the agent browser, so the playing tools are the browser ones: - `browser_play` holds keys while looking, because strafing is two keys at once and aiming while advancing is keys and mouse at once — neither is a sequence of single actions. It always releases what it held. - `browser_mouse_move` takes a delta and primes the origin first: Blink measures movementX against the previous position, so an unprimed turn arrives one step short and the error compounds. - `recordFrames` captures during the action. A screenshot taken after the keys come up is a picture of standing still. - `image_look` reads a contact sheet back, which nothing could do before. And a fix worth its own line: key events no longer carry nativeVirtualKeyCode. It was being given the Windows VK code, which is a different keycode space on macOS — 87 is keypad-5 — so every held key reached the page as Numpad5 and auto-repeated ~20,000 times in 600ms. A game keyed on e.code === "KeyW", which is the usual way, saw nothing. Holding W has probably never worked. Includes concurrent work from other sessions in agent.rs, browser.rs, tools.rs, rpc.rs, approvals.rs, config.rs, graph.rs and guardian.rs: the staged subset would not compile without it, having been verified by building the index in isolation rather than assumed. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NxTEHYdzYKzqMUgb4EyYLh
…oop that lives in core Four ports from `docs/plans/harness-port.md`, all the same shape: a directory of markdown with a `description:` line, plus an index that costs one line per entry until something is invoked. `skills.rs` already implemented that pattern once; three of these are it applied again. **Memory** (`memory.rs`) — one durable fact per file, `~/.cali/memory` plus a per-project directory. The index goes in the volatile tail of the prompt, never into `STATIC_SYSTEM_PROMPT`, which is the shared prompt-cache prefix. Reads are unguarded; `memory_write`/`memory_forget` are not, since they write outside any project and outlive the session. **Hooks** (`hooks.rs`) — `PreToolUse` runs ahead of the permission rules, the mode, and the guardian, because a deterministic rule the user wrote should not be re-litigated by a model. A hook can only ever add a block, never remove one. Every failure proceeds: a broken hook means a check did not run over a call the ordinary gate still sees, and failing closed would let one typo wedge a session. `PostToolUse` attaches its stdout to the tool result, which is where a post-write typecheck belongs. Spawned with `env_clear()` so no API key reaches a command that came out of a config file. **Commands** (`commands.rs`) — `commands/<name>.md` whose body is the prompt. `$ARGUMENTS` and `$1..$9` substitute; an unfilled positional is left as written, because expanding it to nothing turns `costs $5.00` into `costs .00`. **`/loop` moved into core** (`loop_run.rs`) and gained profiles. The driver was ~400 lines in `AgentPanel.tsx` whose cancel flag was a React ref, so a loop died with the tab. `loop_start` now returns as soon as the run is registered and the driver continues detached; the panel only renders the `loop.*` events. The default profile replays the goal verbatim — the old driver rewrote it every iteration into a mandated three-root task graph, so "fix the typo in the README" was answered with a full production. The pipeline is intact as `--aaa`, and its completion defers to `loop_report::validate_completion_readiness`, the same gate a model calling `loop_report_update` must clear. Loop ids were minted from the millisecond alone; two runs started in the same one collided and the second silently replaced the first in the registry, so `loop_stop` reached the wrong run. Now suffixed with a sequence counter. The 30 pipeline tests in `AgentPanel.loop.test.tsx` run unchanged under `--aaa`. Driver behaviour moved to `loop_run`'s own tests, which drive a real run against a scripted in-process provider rather than mocking our own RPC layer. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LUe8nZKANUkSy8fU8MkFb4
… keep a turn going Three follow-ups, one of which fixes a hole in the previous commit. **Rejoining.** Moving the loop driver into core made a run outlive its tab, and then nothing reconnected to it. `listLoopRuns` existed and was called from nowhere: after a reload the run kept working with no one rendering it, every `loop.*` event was discarded for want of an active id, and the composer would start a second loop on the same session. The panel now asks `loop_runs` on mount. With a session open only that session's run is adopted — anything else would stream another chat's turns into the transcript being read — and with none open, which is the ordinary state after a reload because `activeSessionId` is not persisted, the project's run is adopted along with its session. **Subagents from files** (`agents.rs`). `~/.cali/agents/<name>.md`, body as the child's system prompt, optional `tools:` allowlist. `subagent_spawn` always took `role` as a free string; only the composer's hardcoded four made that unreachable. The allowlist is applied in `build_tools`, and the first attempt was wrong in a way worth recording: it filtered the *registered* map, but `build_tools` starts from `core_tool_defs()` and only extends with that map, so the filter removed nothing and a definition claiming its reviewer "cannot write" shipped a child holding `file_write`. An enforcement claim that silently does not enforce is worse than no feature. Now plumbed through `AgentOptions::tool_allowlist` and pinned by a test at the layer that decides. **Stop hooks.** Fired when the agent is about to hand control back; blocking feeds the reason in as the next user turn. This is the seam that lets an autonomous loop be a shell script instead of harness code — verified against a live core, where a six-line hook re-injected twice and let go on DONE. Only top-level turns fire it. A hook written for the main turn does not recognise a subagent's reply, so it blocked every one and drove a single `subagent_spawn` through 199 extra model calls before the turn budget stopped it. Graph nodes run at depth 0 as well, so depth alone is not the test. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01LUe8nZKANUkSy8fU8MkFb4
…nt low The loop could act on a game, capture it, and judge it, but every one of those answered a question about pixels. Nothing could say whether the frame held — the only "fps" in the repo was aaa-fps.json, a genre template — so an aaa run converged on an average score in a rubric with no budget anywhere. game_perf samples the live page for a window and returns frame rate, draw calls, triangles and heap, with an optional budget verdict. It reports fps.low1 first and judges targetFps on it rather than the mean: a page hitching every tenth frame measured p50 120.5 against low1 17.1, and the mean would have called that acceptable. It needs nothing from the game. getContext hands back the same context once one exists, so the live renderer's own draw entry points are wrapped in place — no reload, no build hook — and wrapping the graphics API rather than the engine covers three.js, Babylon, PlayCanvas and raw WebGL alike. Frames that issued no draw call are counted as idle, because rAF keeps ticking at display rate on a black page and would otherwise report a flawless 60. Verified live against a page drawing a known 7 drawArrays per frame: drawCalls.mean 7 and triangles.perFrame 7 exactly, and the patched drawArrays back to native afterwards with no own property left behind. The WebGPU counter is written and unit-pinned but never ran against a real device: headless Chromium exposes no GPURenderPassEncoder, so only the guard that skips it is proven. Flagged in verification.md and in the plan. Also lands two plans: the game_state half of this one, and a shell tool reopening the harness-gaps deferral now that sandbox.rs, hooks.rs and the guardian have shipped the mitigations it was waiting for. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016e1jHKFPaYpnzryU33Sb8K
game_perf shipped as a tool nothing required, so an aaa loop would simply never call it and "AAA" would stay an average score in a rubric. The gate already had a CheckKind::Performance variant it never asked for. validate_completion_readiness now requires a passing Performance check alongside build, play and test. The budget-missed case needs no new rule: game_perf returns a pass/fail verdict, the model records the check with that status, and the existing "any failed check blocks completion" rule stops it. The aaa prompt now tells the model to measure with game_perf and record the numbers, reading fps.low1 rather than fps.mean. This raises the bar rather than lowering it, so it is said out loud: a loop on a surface where the frame budget genuinely cannot be measured will now be refused completion. That is the intended reading — a quality bar with no budget is a taste contest, and the refusal names the missing check. One helper, passing_input, is where tests build a completable report, so the contract change lands in one place and any hand-built report fails loudly. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016e1jHKFPaYpnzryU33Sb8K
`workspace_open` refuses a folder without a package.json or .git, and nothing in core wrote one — so every workspace had to be scaffolded by hand somewhere else before CaliCode could touch it. A starter closes that: a `starter.yaml` manifest beside a `files/` tree that `workspace_create_from_template` writes to disk and then attaches, so the new folder is never orphaned. A starter is not a project template. `store.rs`'s blank|starter|showcase are scene documents the three.js editor owns end to end; a starter is a repository with its own package.json and dev script. Anything with a build belongs in a workspace, because a project document round-trips through one debounced `project_save` and a real game's file tree does not fit through that. Compiled-in starters are `include_str!`d so a packaged app carries them, and a `~/.cali/starters/<id>/` directory shadows a built-in of the same id — the arrangement `graph.rs` already uses for node templates. Two things are deliberately absent rather than half-built: - **Nothing fetches.** A starter is compiled in or already under `~/.cali`, the trust level `~/.cali/commands` and `~/.cali/agents` already have. Cloning a remote repo is arbitrary third-party source arriving because somebody clicked a name, and needs first-use consent keyed on the source the way `approved_project_mcp` gates a project-scoped MCP server. The manifest has no `url:` field, so the half-built version cannot exist. - **Dependencies are not installed.** `npm install` needs the network, and the only sanctioned way to run a command on the user's machine is `terminal.rs`, which is user-initiated by design. `install:` is reported to the client as a string to offer, never spawned. The destination must be absent or empty (merging into a populated directory is how a scaffold silently overwrites work that was never in git), starter paths may not traverse, symlinks are skipped rather than followed, and node_modules is never copied. The built-in `iso-city` starter is the proof the mechanism carries something real: orthographic camera with the polar angle locked at atan(sqrt 2), raycast tile picking against the ground rather than the buildings, one InstancedMesh with swap-remove, and a fixed-timestep sim capped so a backgrounded tab cannot return and lock the page. Driven headlessly before landing: hover reports tiles, clicks place, drag pans without placing, shift-click removes, no console errors. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013CUFpThK67PNcJz16UvH8C
…r tool plumbing Work from a parallel session in this repo, committed together because it is one consistent state and the tree is green with it. - `skills::BUILTIN_SKILLS` compiles skills into the binary and `skills::composed` merges them below every directory, so a user file of the same name shadows one. `goal-loop` moves the 673-token quality loop out of `STATIC_SYSTEM_PROMPT`, where it was billed on every turn of every session, into a skill that costs 69 tokens of description and is loaded only by turns that can act on it. - A `deny` permission rule now removes the tool from what the model is sent, making it the supported way to stop paying for a family rather than only to refuse it. - `graph.rs` repairs judge dependency-sheet ordering, which had left `judge_receives_all_dependency_sheets_in_stable_order` red on the branch. - Browser tool plumbing in `useBrowserTools`. Verified: cargo test 1075 passed, 0 failed. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013CUFpThK67PNcJz16UvH8C
…passing performance check its own rule demands Two failures that had never been seen, because CI had not run on this branch before. `computer.rs`'s five `#[ignore]`d live and diagnostic tests call CoreGraphics and SkyLight helpers that are defined only under `cfg(target_os = "macos")`. The tests themselves carried no such gate, so `clippy --all-targets` on the Linux runner could not resolve `frontmost_app_name`, `click_via`, `mouse_events`, `post_via_skylight` or `scroll_by`. They are macOS-only by nature — the whole module is — so the gate goes on the tests. `reports.spec.ts` built a final iteration whose performance check was `skipped`. `validate_completion_readiness` requires a *passing* one, which is the point of "a run that never timed a frame cannot call itself done": the fixture described a report that can no longer reach `completed`. The fixture is what changed, not the rule — the rule is the intended behaviour and the fixture predates it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013CUFpThK67PNcJz16UvH8C
Gating the macOS-only tests removed the last Linux-side users of four items, which `-D dead-code` then rejected. These were always broken on Linux; the unresolved-function errors simply aborted compilation before the lint could run, so each fix uncovers the next layer. - `MAX_CAPTURE_EDGE` and `keycode` are used only by `capture_window` and `press_key`, both macOS-only. Gated to match. - The `not(macos)` stubs for `click_at` and `post_to_pid` are removed: every caller is inside a macOS-gated function or test, so on Linux nothing reached them. Checked by hand that the plain `#[test]` helpers still have Linux callers — `map_to_screen`, `remember_capture` and `capture_sizes` are used by `image_coordinates_map_onto_the_window` and its neighbours, which carry no target gate. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013CUFpThK67PNcJz16UvH8C
…ks ratio `workflow.spec.ts` polled until the `editor_attach` *request* was seen on the wire, then immediately issued `editor_tool_call`. Core registers the session's editor owner while handling that attach, so the tool call could arrive before an owner existed and hang until the 60s test timeout rather than failing with a reason. `requestfinished` fires once the response is back, which is the point at which the attach has actually been applied. The routing assertions are unchanged — this is synchronisation, not a loosened check. `reports.spec.ts`'s Checks metric is `checks_passed / (passed + failed + skipped)` (`refresh_totals`). Making the final performance check passed rather than skipped moves it from the skipped bucket to the passed one, so the header reads 5/6. "2h 35m" is unaffected: worked duration sums iteration durations, not check ones. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_013CUFpThK67PNcJz16UvH8C
In auto mode the lever schema was gated on `def.access`. That field is authoritative for core tools but is the fail-closed `Guarded` default for every client-registered tool, since `tool_register` sends no access field. The gate itself classifies with `is_destructive`, so the lever offered itself on every `editor_*` tool while guarding none of them. Two independent fixes on the same mismatch: - Gate the lever schema on `is_destructive` so it is offered exactly where the gate guards, with a regression test for a client-registered reader. - Stop the lever being used as a check-in: keep the fail-closed escalation but render an empty question by naming the tool, and tell the model not to use the lever to read, inspect, validate, test, or delegate.
duolahypercho
force-pushed
the
harness-port
branch
from
August 18, 2026 02:20
0371bf7 to
fc213df
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This lands the remaining harness work on top of the current
mainbranch:/loopwith standard and AAA profiles, resume/stop support, evidence gates, and durable reportsIntegration
The branch was rebased onto
mainafter PRs #45, #46, #47, and #49 landed. Patch-equivalent asset, Electron-shell, approval-rendering, and concurrent-editor E2E changes were skipped in favor of the versions already onmain.Conflict resolution was audited against both the original
harness-porthead and currentmain. That audit caught and restored the guardian-model selector, permission-mode copy, Electron browser-attach regression coverage, and repository instructions before this final push.Verification
cargo fmt --checkcargo clippy --all-targets -- -D warningscargo test: 1,076 passed, 17 ignored; 2 shutdown tests passedbrowser::tests::live_attachregression: passednpx tsc -b --noEmitpnpm test: 941 passed across 75 filespnpm buildpnpm test:e2e: 71 passed, 0 failedgit diff --checkNo debug instrumentation or speculative App/SSE changes are included.