From b86ac47094c8b5cac8e2a931ad92b96eb9c4772c Mon Sep 17 00:00:00 2001 From: logan Date: Sun, 16 Aug 2026 03:03:52 -0400 Subject: [PATCH 1/4] fix: a fresh arm directory runs the resume check in silence (#109) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The first real review arm on issue #109 printed `no matches found` from the resume check and collected nothing. The message and the abort are two events, and reading them as one hid both. The resume check asks which sidecars an arm already holds, with `ls` over a glob. A fresh arm holds none, so zsh answers NOMATCH: the substitution fails, the line prints, and the run carries on with `existing` empty, which is the correct reading of an empty arm. The arm then reached its first live call, and the shell was killed by a signal during it — before the line that removes an empty `.err`, and without the EXIT trap, which is why the directory held one zero-byte `.err` and no manifest. A stand-in `claude` reproduces each half: the fixed runner completes a fresh arm clean, and a stand-in that signals its own process group leaves exactly the directory the operator saw. The glob is now the `(N)` null-glob qualifier, so no match gives no words. A check that passes has to be silent as well as harmless, because the arm spends a live call under it. The refusal is untouched: an arm resumed under a changed system prompt or a changed rule set still exits 2 rather than mixing two conditions under one name. Nothing exercised the runner. `test/bench-helpers.js` builds an arm by writing the files an arm holds, which is every step except the ones only the runner takes, so the smoke path could not see this. `test/run-sh.test.js` drives `bench/run.sh` end to end over a stand-in `claude`, on a fresh directory and then on a changed configuration. It skips where zsh is absent and it reaches no model. `bench/README.md` now states what the calling shell supplies for a real run, including the credential route, and the CI comment that said the runner is never invoked is corrected. --- .github/workflows/ci.yml | 6 ++- AGENTS.md | 6 +++ bench/README.md | 14 +++--- bench/run.sh | 13 +++++- test/run-sh.test.js | 94 ++++++++++++++++++++++++++++++++++++++++ 5 files changed, 125 insertions(+), 8 deletions(-) create mode 100644 test/run-sh.test.js diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index dcd6b0f..3de67b6 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -90,8 +90,10 @@ jobs: - run: npm ci # `npm run check` is the same set of checks the ubuntu job runs as named - # steps. None of them shells out, so nothing here needs zsh. - # `bench/run.sh` does, and CI never invokes the runner. + # steps. One of them shells out: `test/run-sh.test.js` drives + # `bench/run.sh` over a stand-in `claude`, so a host with zsh exercises + # the runner's control flow and a host without one skips those two tests. + # Nothing here ever reaches a model. - name: Every check run: npm run check diff --git a/AGENTS.md b/AGENTS.md index fec7ee3..0f79cf3 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -847,6 +847,12 @@ Do not read a green pipeline as coverage of this one. `test/gfm-render.test.js` is the only evidence that the grammar admits the prose a reader admits. Read a green catalogue run as evidence about the block path, and not about this one. +- **The runner is driven over a stand-in, and only where zsh is installed.** + `test/run-sh.test.js` runs `bench/run.sh` end to end with a fake `claude` on + `PATH`, which is what a fresh arm directory needed and no fixture-built arm + could give it. It says nothing about the real CLI, its flags, or the + credential the calling shell supplies, and it skips entirely on a host with + no zsh. Read a green run as evidence about the runner's control flow. - **An editorial stamp is a claim, and the suite checks its form alone.** `test/editorial.test.js` covers the record's table, the day, the digest and both notes. Nothing can tell a row a person wrote after reading from a row an diff --git a/bench/README.md b/bench/README.md index 9e88626..6f249a3 100644 --- a/bench/README.md +++ b/bench/README.md @@ -11,11 +11,15 @@ reproducible and artificial. The field half is real and uncontrolled. Fixed scenarios, fresh context, several runs each, one variable at a time. -`run.sh` needs **zsh** and the `claude` CLI on `PATH`. Neither is a dependency of -this package, and nothing in `npm run check` invokes the runner, so continuous -integration never exercises this half. A Linux container without zsh will fail -every command below before the harness starts. The scorer is plain Node and runs -anywhere. +`run.sh` needs **zsh** and the `claude` CLI on `PATH`. The calling shell also +supplies the credential, because the runner builds no environment and reads no +value: a real arm authenticates by the routes issue #77 ranks, which are +`CLAUDE_CODE_OAUTH_TOKEN` for a subscription ahead of `ANTHROPIC_API_KEY`. None +of that is a dependency of this package. A Linux container without zsh will fail +every command below before the harness starts. `test/run-sh.test.js` drives the +runner over a stand-in `claude` and skips where zsh is absent, so what +continuous integration exercises is the runner's own control flow and never a +model call. The scorer is plain Node and runs anywhere. ``` bench/run.sh control # no guidance at all diff --git a/bench/run.sh b/bench/run.sh index 0773941..7fa303f 100755 --- a/bench/run.sh +++ b/bench/run.sh @@ -134,7 +134,18 @@ trap "$CLEAN_UP" EXIT # old samples and generates only the missing ones, so the cell silently holds # two conditions. The arm directory name is not a fingerprint. Compare against # what is already there and refuse rather than resume. -existing="$(ls "$HERE/out/$ARM"/*.meta 2>/dev/null | head -1)" +# +# The comparison reads whatever sidecars are there, and a FRESH arm has none. +# `ls` over an unmatched glob is how that was asked, and zsh answers NOMATCH: +# the substitution failed, `no matches found` printed over a run that then +# proceeded correctly, and an operator read the first real review arm as broken +# on its first line of work. The message is what cost the run, so a check that +# passes has to be silent as well as harmless — the arm went on to spend a live +# call under it. `(N)` is the null-glob qualifier, so no match gives no words. The +# first element is the sidecar to compare against, and an empty array leaves +# `existing` empty, which is the state that skips the block below. +sidecars=("$HERE/out/$ARM"/*.meta(N)) +existing="${sidecars[1]}" if [ -n "$existing" ]; then was_system="$(sed -n 's/.*system_sha=\([^ ]*\).*/\1/p' "$existing")" was_rules="$(sed -n 's/.*rules=\([^ ]*\) .*/\1/p' "$existing")" diff --git a/test/run-sh.test.js b/test/run-sh.test.js new file mode 100644 index 0000000..d519a91 --- /dev/null +++ b/test/run-sh.test.js @@ -0,0 +1,94 @@ +import { test } from 'node:test'; +import assert from 'node:assert/strict'; +import { execFile } from 'node:child_process'; +import fs from 'node:fs/promises'; +import os from 'node:os'; +import path from 'node:path'; + +/** + * The runner itself, driven end to end over a stand-in `claude`. + * + * Everything else that touches an arm builds one by writing the FILES an arm + * holds — `test/bench-helpers.js` does exactly that, and so the runner that + * produces them had no exercise anywhere. The first real review arm then hit a + * failure on its first line of work that no smoke run could have seen: a fresh + * arm directory matches no sidecar, and the resume check asked the question + * with a glob whose NOMATCH zsh reports. Building the output skips the runner, + * so the smoke path and the real path diverged on the one step only the runner + * takes. + * + * The stand-in echoes one fixed JSON run, so this costs no model call. It is + * the same trick `bench/review-arms.mjs` gets from an injected `git`. + * + * The runner writes under `bench/out/`, which is where it always writes, so + * these arms are named for this test and removed after it. CI does not run + * this: `bench/run.sh` needs zsh, and a host without it skips rather than + * fails, which `bench/README.md` already states for every command in that + * directory. + */ + +const REPO = path.dirname(import.meta.dirname); +const RUN = path.join(REPO, 'bench', 'run.sh'); + +const STANDIN = `#!/bin/sh +if [ "$1" = "--version" ]; then echo "2.1.220 (Claude Code)"; exit 0; fi +cat <<'JSON' +{"is_error":false,"result":"A stand-in answer.","modelUsage":{"m-1":{"outputTokens":11}}} +JSON +`; + +const hasZsh = await fs.stat('/bin/zsh').then(() => true, () => false); + +function runArm(args, binDir) { + return new Promise((resolve) => { + execFile('/bin/zsh', [RUN, ...args], { + cwd: REPO, env: { ...process.env, PATH: `${binDir}${path.delimiter}${process.env.PATH}` }, + }, (err, stdout, stderr) => resolve({ code: err ? err.code ?? 1 : 0, stdout, stderr })); + }); +} + +async function scaffold(t, arm) { + const root = await fs.mkdtemp(path.join(os.tmpdir(), 'sw-run-')); + const armDir = path.join(REPO, 'bench', 'out', arm); + t.after(() => Promise.all([ + fs.rm(root, { recursive: true, force: true }), + fs.rm(armDir, { recursive: true, force: true }), + ])); + const binDir = path.join(root, 'bin'); + const prompts = path.join(root, 'prompts'); + await fs.mkdir(binDir); + await fs.mkdir(prompts); + await fs.writeFile(path.join(binDir, 'claude'), STANDIN, { mode: 0o755 }); + await fs.writeFile(path.join(prompts, 'pr-1-r1.txt'), 'Say something.\n'); + return { armDir, binDir, prompts }; +} + +test('a fresh arm directory runs clean, and writes a sample, a sidecar and a manifest', + { skip: hasZsh ? false : 'zsh is not installed, and bench/run.sh is zsh' }, async (t) => { + const arm = 'test-fresh-arm'; + const { armDir, binDir, prompts } = await scaffold(t, arm); + const run = await runArm([arm, '--prompts', prompts, '--reps', '2'], binDir); + assert.equal(run.code, 0, run.stderr); + // The failure this test exists for. It printed here and the run carried on, + // so the arm read as broken to the person watching it. + assert.ok(!run.stderr.includes('no matches found'), run.stderr); + const held = (await fs.readdir(armDir)).sort(); + assert.deepEqual(held, [ + 'arm-manifest.json', + 'pr-1-r1-1.txt', 'pr-1-r1-1.txt.meta', + 'pr-1-r1-2.txt', 'pr-1-r1-2.txt.meta', + ]); + }); + +test('an arm resumed under a changed configuration is still refused', + { skip: hasZsh ? false : 'zsh is not installed, and bench/run.sh is zsh' }, async (t) => { + // The refusal the glob was asking for. A fresh directory must not reach it, + // and a collected one must, or an arm silently holds two conditions. + const arm = 'test-resume-arm'; + const { binDir, prompts } = await scaffold(t, arm); + assert.equal((await runArm([arm, '--prompts', prompts, '--reps', '2'], binDir)).code, 0); + const again = await runArm( + [arm, '--prompts', prompts, '--reps', '2', '--system', 'bench/review-contract.md'], binDir); + assert.equal(again.code, 2); + assert.match(again.stderr, /Use a new arm name/); + }); From 073063a56f99e71ecb2649493099b97b76905b48 Mon Sep 17 00:00:00 2001 From: logan Date: Sun, 16 Aug 2026 03:10:16 -0400 Subject: [PATCH 2/4] test: an arm this test collects is named for nothing else (#122) `run.sh` writes under its own `bench/out/`, so a literal arm name here is a directory in the operator's tree. Two test processes on one checkout collide there, and anything already standing at the name would be resumed by the fresh -arm test and then removed by its cleanup. The name now comes from the per-test temporary directory, which is unique to the run, and the test asserts it is a name `arm-manifest.mjs` accepts. Two more from reading the file back. zsh is asked for by name through `PATH` rather than at `/bin/zsh`, which is macOS's path and would have skipped the whole file on Linux while `execFile` used it anyway. And the header no longer says CI does not run this, which the corrected workflow comment already contradicts. --- COMMIT_MSG_TMP.txt | 14 ++++++++++++++ test/run-sh.test.js | 41 ++++++++++++++++++++++++++++------------- 2 files changed, 42 insertions(+), 13 deletions(-) create mode 100644 COMMIT_MSG_TMP.txt diff --git a/COMMIT_MSG_TMP.txt b/COMMIT_MSG_TMP.txt new file mode 100644 index 0000000..52eb5b3 --- /dev/null +++ b/COMMIT_MSG_TMP.txt @@ -0,0 +1,14 @@ +test: an arm this test collects is named for nothing else (#122) + +`run.sh` writes under its own `bench/out/`, so a literal arm name here is a +directory in the operator's tree. Two test processes on one checkout collide +there, and anything already standing at the name would be resumed by the fresh +-arm test and then removed by its cleanup. The name now comes from the +per-test temporary directory, which is unique to the run, and the test asserts +it is a name `arm-manifest.mjs` accepts. + +Two more from reading the file back. zsh is asked for by name through `PATH` +rather than at `/bin/zsh`, which is macOS's path and would have skipped the +whole file on Linux while `execFile` used it anyway. And the header no longer +says CI does not run this, which the corrected workflow comment already +contradicts. diff --git a/test/run-sh.test.js b/test/run-sh.test.js index d519a91..d1c474c 100644 --- a/test/run-sh.test.js +++ b/test/run-sh.test.js @@ -5,6 +5,8 @@ import fs from 'node:fs/promises'; import os from 'node:os'; import path from 'node:path'; +import { NAME } from '../bench/arm-manifest.mjs'; + /** * The runner itself, driven end to end over a stand-in `claude`. * @@ -20,11 +22,11 @@ import path from 'node:path'; * The stand-in echoes one fixed JSON run, so this costs no model call. It is * the same trick `bench/review-arms.mjs` gets from an injected `git`. * - * The runner writes under `bench/out/`, which is where it always writes, so - * these arms are named for this test and removed after it. CI does not run - * this: `bench/run.sh` needs zsh, and a host without it skips rather than - * fails, which `bench/README.md` already states for every command in that - * directory. + * A host with zsh runs this and a host without one skips it, which is the + * disposition `bench/README.md` already gives every command in that directory. + * zsh is asked for by name through `PATH` rather than at `/bin/zsh`, because + * that path is macOS's and Linux keeps it elsewhere — testing the wrong one + * skips the whole file on a host that could have run it. */ const REPO = path.dirname(import.meta.dirname); @@ -37,18 +39,33 @@ cat <<'JSON' JSON `; -const hasZsh = await fs.stat('/bin/zsh').then(() => true, () => false); +const hasZsh = await new Promise((resolve) => { + execFile('zsh', ['-c', ':'], (err) => resolve(!err)); +}); function runArm(args, binDir) { return new Promise((resolve) => { - execFile('/bin/zsh', [RUN, ...args], { + execFile('zsh', [RUN, ...args], { cwd: REPO, env: { ...process.env, PATH: `${binDir}${path.delimiter}${process.env.PATH}` }, }, (err, stdout, stderr) => resolve({ code: err ? err.code ?? 1 : 0, stdout, stderr })); }); } -async function scaffold(t, arm) { +/** + * The arm name comes from the temporary directory, because the arm directory + * does not. + * + * `run.sh` writes under its own `bench/out/`, and that is the operator's tree + * where real samples live. A name written here as a literal is a name this + * test would resume if anything already stood at it, and then delete on the + * way out — two processes sharing one checkout is enough to produce it. The + * name `mkdtemp` gives is unique to this run and is already a name + * `arm-manifest.mjs` accepts. + */ +async function scaffold(t) { const root = await fs.mkdtemp(path.join(os.tmpdir(), 'sw-run-')); + const arm = path.basename(root); + assert.ok(NAME.test(arm), `${arm} must be a name the arm manifest accepts`); const armDir = path.join(REPO, 'bench', 'out', arm); t.after(() => Promise.all([ fs.rm(root, { recursive: true, force: true }), @@ -60,13 +77,12 @@ async function scaffold(t, arm) { await fs.mkdir(prompts); await fs.writeFile(path.join(binDir, 'claude'), STANDIN, { mode: 0o755 }); await fs.writeFile(path.join(prompts, 'pr-1-r1.txt'), 'Say something.\n'); - return { armDir, binDir, prompts }; + return { arm, armDir, binDir, prompts }; } test('a fresh arm directory runs clean, and writes a sample, a sidecar and a manifest', { skip: hasZsh ? false : 'zsh is not installed, and bench/run.sh is zsh' }, async (t) => { - const arm = 'test-fresh-arm'; - const { armDir, binDir, prompts } = await scaffold(t, arm); + const { arm, armDir, binDir, prompts } = await scaffold(t); const run = await runArm([arm, '--prompts', prompts, '--reps', '2'], binDir); assert.equal(run.code, 0, run.stderr); // The failure this test exists for. It printed here and the run carried on, @@ -84,8 +100,7 @@ test('an arm resumed under a changed configuration is still refused', { skip: hasZsh ? false : 'zsh is not installed, and bench/run.sh is zsh' }, async (t) => { // The refusal the glob was asking for. A fresh directory must not reach it, // and a collected one must, or an arm silently holds two conditions. - const arm = 'test-resume-arm'; - const { binDir, prompts } = await scaffold(t, arm); + const { arm, binDir, prompts } = await scaffold(t); assert.equal((await runArm([arm, '--prompts', prompts, '--reps', '2'], binDir)).code, 0); const again = await runArm( [arm, '--prompts', prompts, '--reps', '2', '--system', 'bench/review-contract.md'], binDir); From 20d20153353b8a84566dae7e77e6fbb57c6b4f7a Mon Sep 17 00:00:00 2001 From: logan Date: Sun, 16 Aug 2026 03:33:35 -0400 Subject: [PATCH 3/4] docs: the arm that printed the glob error was healthy, and finished (#109) New evidence retracts the diagnosis this branch was written under. The first review arm completed about three hours after it launched: eight samples, both manifests covering their plans, exit 0. So the zero-byte `.err` and the absent manifest were a snapshot of a call still OPEN, and the reproduction that matched them byte for byte was evidence of that state and not of a kill. A signal was named over an artifact that not-being-finished-yet produces, and the benign reading was never ruled out. The fix stands and the reason for it changes. The NOMATCH was cosmetic noise on a working run, and the cost was the reading: an arm that prints a failure and then says nothing for hours cannot be told from a dead one, which is what the unexercised runner left nobody able to check. A passing check has to be silent, because silence is what the next line of output is read against. `bench/run.sh` and `test/run-sh.test.js` carry the corrected account. `bench/README.md` says an arm is slow and silence is not failure, with the duration marked unaudited. The blind-spot entry in AGENTS.md now names duration as the thing a stand-in cannot measure. --- AGENTS.md | 4 ++++ COMMIT_MSG_TMP.txt | 30 ++++++++++++++++++------------ bench/README.md | 5 +++++ bench/run.sh | 17 +++++++++++------ test/run-sh.test.js | 18 ++++++++++-------- 5 files changed, 48 insertions(+), 26 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index 0f79cf3..efaf7cd 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -853,6 +853,10 @@ Do not read a green pipeline as coverage of this one. could give it. It says nothing about the real CLI, its flags, or the credential the calling shell supplies, and it skips entirely on a host with no zsh. Read a green run as evidence about the runner's control flow. + It says nothing about DURATION either, and that is the reading this hole + cost. The stand-in answers at once, where a real arm is quiet for hours, so + an arm that had merely not finished was read here as an arm that had died — + and the artifacts of the two are identical while the call is open. - **An editorial stamp is a claim, and the suite checks its form alone.** `test/editorial.test.js` covers the record's table, the day, the digest and both notes. Nothing can tell a row a person wrote after reading from a row an diff --git a/COMMIT_MSG_TMP.txt b/COMMIT_MSG_TMP.txt index 52eb5b3..03d0601 100644 --- a/COMMIT_MSG_TMP.txt +++ b/COMMIT_MSG_TMP.txt @@ -1,14 +1,20 @@ -test: an arm this test collects is named for nothing else (#122) +docs: the arm that printed the glob error was healthy, and finished (#109) -`run.sh` writes under its own `bench/out/`, so a literal arm name here is a -directory in the operator's tree. Two test processes on one checkout collide -there, and anything already standing at the name would be resumed by the fresh --arm test and then removed by its cleanup. The name now comes from the -per-test temporary directory, which is unique to the run, and the test asserts -it is a name `arm-manifest.mjs` accepts. +New evidence retracts the diagnosis this branch was written under. The first +review arm completed about three hours after it launched: eight samples, both +manifests covering their plans, exit 0. So the zero-byte `.err` and the absent +manifest were a snapshot of a call still OPEN, and the reproduction that +matched them byte for byte was evidence of that state and not of a kill. A +signal was named over an artifact that not-being-finished-yet produces, and +the benign reading was never ruled out. -Two more from reading the file back. zsh is asked for by name through `PATH` -rather than at `/bin/zsh`, which is macOS's path and would have skipped the -whole file on Linux while `execFile` used it anyway. And the header no longer -says CI does not run this, which the corrected workflow comment already -contradicts. +The fix stands and the reason for it changes. The NOMATCH was cosmetic noise +on a working run, and the cost was the reading: an arm that prints a failure +and then says nothing for hours cannot be told from a dead one, which is what +the unexercised runner left nobody able to check. A passing check has to be +silent, because silence is what the next line of output is read against. + +`bench/run.sh` and `test/run-sh.test.js` carry the corrected account. +`bench/README.md` says an arm is slow and silence is not failure, with the +duration marked unaudited. The blind-spot entry in AGENTS.md now names +duration as the thing a stand-in cannot measure. diff --git a/bench/README.md b/bench/README.md index 6f249a3..a773b7c 100644 --- a/bench/README.md +++ b/bench/README.md @@ -21,6 +21,11 @@ runner over a stand-in `claude` and skips where zsh is absent, so what continuous integration exercises is the runner's own control flow and never a model call. The scorer is plain Node and runs anywhere. +**An arm is slow, and silence is not failure.** The first review arm on issue +#109 took about three hours to collect eight samples, unaudited, and it printed +nothing between them. Read a running arm by the files it has written, and give +a claim that one died the evidence such a claim needs. + ``` bench/run.sh control # no guidance at all bench/run.sh with-skill --system skills/craft/compressed-deliberation/SKILL.md diff --git a/bench/run.sh b/bench/run.sh index 7fa303f..c853cd7 100755 --- a/bench/run.sh +++ b/bench/run.sh @@ -137,12 +137,17 @@ trap "$CLEAN_UP" EXIT # # The comparison reads whatever sidecars are there, and a FRESH arm has none. # `ls` over an unmatched glob is how that was asked, and zsh answers NOMATCH: -# the substitution failed, `no matches found` printed over a run that then -# proceeded correctly, and an operator read the first real review arm as broken -# on its first line of work. The message is what cost the run, so a check that -# passes has to be silent as well as harmless — the arm went on to spend a live -# call under it. `(N)` is the null-glob qualifier, so no match gives no words. The -# first element is the sidecar to compare against, and an empty array leaves +# the substitution fails and `no matches found` prints, over a run that then +# proceeds correctly with `existing` empty. The first review arm on issue #109 +# printed it and went on to collect all eight of its samples three hours later. +# So the message cost nobody a sample and it cost the operator the run anyway: +# an arm that prints a failure on its first line of work and then says nothing +# for hours cannot be told from a dead one, and it was read as dead. A check +# that passes has to be SILENT as well as harmless, because silence is the only +# thing the next line of output is measured against. +# +# `(N)` is the null-glob qualifier, so no match gives no words. The first +# element is the sidecar to compare against, and an empty array leaves # `existing` empty, which is the state that skips the block below. sidecars=("$HERE/out/$ARM"/*.meta(N)) existing="${sidecars[1]}" diff --git a/test/run-sh.test.js b/test/run-sh.test.js index d1c474c..24252ad 100644 --- a/test/run-sh.test.js +++ b/test/run-sh.test.js @@ -12,12 +12,13 @@ import { NAME } from '../bench/arm-manifest.mjs'; * * Everything else that touches an arm builds one by writing the FILES an arm * holds — `test/bench-helpers.js` does exactly that, and so the runner that - * produces them had no exercise anywhere. The first real review arm then hit a - * failure on its first line of work that no smoke run could have seen: a fresh - * arm directory matches no sidecar, and the resume check asked the question - * with a glob whose NOMATCH zsh reports. Building the output skips the runner, - * so the smoke path and the real path diverged on the one step only the runner - * takes. + * produces them had no exercise anywhere. The first real review arm on issue + * #109 then printed `no matches found` on its first line of work, from a + * resume check asking a glob about a fresh directory zsh answers NOMATCH for. + * That arm was healthy and finished every sample hours later. Nobody could say + * so at the time, which is what the hole actually cost: with the runner + * unexercised, a slow run and a dead one produce the same evidence, and the + * benign reading is the one that goes unconsidered. * * The stand-in echoes one fixed JSON run, so this costs no model call. It is * the same trick `bench/review-arms.mjs` gets from an injected `git`. @@ -85,8 +86,9 @@ test('a fresh arm directory runs clean, and writes a sample, a sidecar and a man const { arm, armDir, binDir, prompts } = await scaffold(t); const run = await runArm([arm, '--prompts', prompts, '--reps', '2'], binDir); assert.equal(run.code, 0, run.stderr); - // The failure this test exists for. It printed here and the run carried on, - // so the arm read as broken to the person watching it. + // The line this test exists for. The run carried on correctly under it, + // which is why a passing check has to be silent: the operator has nothing + // but the output to tell a working arm from a stopped one. assert.ok(!run.stderr.includes('no matches found'), run.stderr); const held = (await fs.readdir(armDir)).sort(); assert.deepEqual(held, [ From f105963d8e28a9ef4f8e3e32b1a1d67a17b1633b Mon Sep 17 00:00:00 2001 From: logan Date: Sun, 16 Aug 2026 03:34:19 -0400 Subject: [PATCH 4/4] chore: a commit message file is not a tracked file `git add -A` staged the scratch file two of these commits were written from. The net diff against main no longer carries it. --- COMMIT_MSG_TMP.txt | 20 -------------------- 1 file changed, 20 deletions(-) delete mode 100644 COMMIT_MSG_TMP.txt diff --git a/COMMIT_MSG_TMP.txt b/COMMIT_MSG_TMP.txt deleted file mode 100644 index 03d0601..0000000 --- a/COMMIT_MSG_TMP.txt +++ /dev/null @@ -1,20 +0,0 @@ -docs: the arm that printed the glob error was healthy, and finished (#109) - -New evidence retracts the diagnosis this branch was written under. The first -review arm completed about three hours after it launched: eight samples, both -manifests covering their plans, exit 0. So the zero-byte `.err` and the absent -manifest were a snapshot of a call still OPEN, and the reproduction that -matched them byte for byte was evidence of that state and not of a kill. A -signal was named over an artifact that not-being-finished-yet produces, and -the benign reading was never ruled out. - -The fix stands and the reason for it changes. The NOMATCH was cosmetic noise -on a working run, and the cost was the reading: an arm that prints a failure -and then says nothing for hours cannot be told from a dead one, which is what -the unexercised runner left nobody able to check. A passing check has to be -silent, because silence is what the next line of output is read against. - -`bench/run.sh` and `test/run-sh.test.js` carry the corrected account. -`bench/README.md` says an arm is slow and silence is not failure, with the -duration marked unaudited. The blind-spot entry in AGENTS.md now names -duration as the thing a stand-in cannot measure.