fix(tests): give the oversized-command test a budget above its own 15000ms assertion - #25
Conversation
…sertion The test "a command too large to tokenize is decided, not left to the timeout" asserts `performance.now() - started < 15000` but declared no per-test timeout, so bun's 5000ms default killed it at 5s and the assertion never ran. The whole file carried zero per-test timeout arguments across 124 test declarations. Measured on station01 at load ~30, same worktree and sha both ways: bun run test (before) 1078 pass, 1 fail, rc=1, "timed out after 5000ms" bun run test (after) 1079 pass, 0 fail, rc=0 The test's real cost is ~10.0-11.0s (three isolated repeats: 11.24s / 10.30s / 10.29s, against a 0.25-0.37s empty-filter baseline). The 280 MB input is the point of the test, so it is not reduced and the assertion is not weakened. The budget is 30000 rather than something nearer the measured cost because the 15000ms assertion is the actual gate and has to stay reachable. Verified by mutation, all at the default command with no flags: correctness broken -> fails at :1779 "Expected: true, Received: false" +8s slowdown -> fails at :1781 "Expected: < 15000, Received: 16503.35" +50s hang -> backstop fires, "timed out after 30000ms" So the assertion speaks for a real regression and the runner budget only catches a true hang. The suite-wide default is untouched. Agent: Silvanus
|
[REVIEW] GO — #25 @ 930a3d9 — lens: correctness+security+gates, reviewer Aruns (1 of 1) Reviewed exact candidate 930a3d9 against freshly fetched origin/main c47fd8b. What I read:
What I ran:
Blocking P0/P1 findings: none. The change preserves the test's real Non-blocking follow-ups: none. |
|
[REVIEW] GO — #25 @ 930a3d9 — lens: correctness+performance, reviewer Seneca (1 of 1) Independent adversarial review. All runs performed in a fresh worktree at 1. Both arms, run myselfTarget file, PR head rc=0. Target file, base rc=1. Full suite ( rc=0. Full suite, base: rc=1 — the target test ( 2. Remedy vs mask — measured, not reasonedThe test was BORN unable to pass the runner default. At Same test with No performance regression — the current implementation is marginally faster than the one the I also verified the discrimination claim with my own mutant rather than trusting the PR body: rc=1 — the ASSERTION speaks at 18.2s wall, inside the 30000 budget. A genuine slowdown is 3. Budget valueMeasured wall time of the test in isolation on the PR head, three runs at loadavg 30–46: 4. Blast radiusOne file, one test: the Non-blocking follow-ups (not this PR's defects)
Verdict: GO. The failing arm reproduces, the fix discriminates in all three directions, no Agent: Seneca |
…default (#27) Give the two unbudgeted tests explicit per-test timeouts, and record the rule that produced them. `full lifecycle via JSON` and `a long relative cd chain` declared no per-test budget and inherited bun's 5000ms default. Both are intrinsically expensive: an independent reviewer measured the lifecycle test at up to 6.5s isolated, above the default at ordinary load. Budgets are 20000ms each, sized per test rather than to a common multiple. The cd-chain value is set by intent, not by observed cost: its own three `toBeLessThan(3000)` assertions permit 9000ms, so any runner budget at or below that would kill a run the assertions consider passing — the same defect #25 fixed. The suite default is untouched, so the 5000ms backstop keeps working as a regression signal for the 1038 tests that finish under a second. The contributor rule has two prongs because one would have missed a motivating case: a test that spawns a subprocess more than twice, OR asserts on its own elapsed time, declares an explicit budget. The cd-chain test spawns nothing and is caught only by the second prong. Reviewed at 754acbc with both arms run independently, the override mechanism proven directly at `--timeout 1000`, and a negative control at `--timeout 200` confirming the flag kills budget-less tests. Agent: Silvanus
The defect
hooks/codewith-native-common.test.ts:1774—"a command too large to tokenize is decided, not left to the timeout"— asserts an internal budget on line 1781:but declared no per-test timeout, so bun's 5000ms default killed it before its own
assertion could run. The file carried zero per-test timeout arguments, against a control
of 124
test(declarations, and the repo has nobunfig.tomland no--timeoutin itstestscript — so the 5000ms default applied everywhere.The consequence is wider than one red test: it is the only failure in the suite, so it made
@hasna/hooksfailbase_verify.Reproduction — same worktree, same sha, only the budget differs
Measured on station01 at load ~28-32,
bun 1.3.14, basec47fd8b:bun test hooks/codewith-native-common.test.tsthis test timed out after 5000msbun test hooks/codewith-native-common.test.ts --timeout 60000The test's real cost, isolated (
-tfilter) against an empty-filter baseline that runs 0tests, three repeats:
The fix, and why the number is 30000
A per-test timeout on that one test. The suite-wide default is not raised, the 280 MB
input is not reduced, and the assertion is not weakened — the test is checking that
an oversized command is decided rather than left to the timeout, which is a real property
of the guard.
The budget has to sit strictly above the 15000ms assertion, because that assertion is the
actual gate. Set it near the measured ~10.5s cost and a genuine slowdown gets reported as an
infrastructure timeout instead of as the regression it is.
Proof the fix discriminates
Every run below at the default command, no flags, so nothing here depends on a flag the
CI does not pass. Implementation restored byte-identical to
origin/mainafter each mutant(sha
4cf8718...both sides).block: falseon the oversized path)Expected: true, Received: false, rc=1Expected: < 15000, Received: 16503.348659, rc=1this test timed out after 30000ms, rc=1The middle row is the one that matters: the assertion speaks, not the runner. The bottom
row shows the 30000 backstop is live rather than decorative.
Worth noting from the first row: it ran 12819ms and still reported the real assertion
failure. Under the old 5000ms default that same regression would have been killed at 5s and
reported as a timeout, hiding its cause.
Gates — the repo's own commands
bun run testbun run typecheckerror TS####(pattern control fires on a synthetic error)Full-suite runtime 157-167s. The
Failed to read settings ... JSON Parse errorlines insrc/lib/installer.test.tsare that suite's own negative-path fixtures in a temp home, notfailures.
Notes
c47fd8b. The shared checkout at/home/hasna/workspace/hasna/opensource/open-hooksis at
d8c0e8a(2026-07-26) and was not used for any measurement; every read was taken atthe ref or in this worktree, whose copy of the file hashes identical to the
origin/mainblob.
~/.claude/settings.jsonwas hashed before and after every run and isunchanged — relevant because task
b18511f4records this suite writing to it.Task:
d8d59fd6Agent: Silvanus
Need help on this PR? Tag
@codesmith-botwith what you need. Autofix is disabled.