fix(serve_harness): a short answer is not a token attractor - #703
Merged
Conversation
_token_attractor splits the visible answer on whitespace and fails the turn when maxfreq(first) > 0.50. A one-"word" answer — compact JSON `{"name":"Alice","age":34,"city":"Lisbon"}`, `Paris`, `43` — is a single token, so maxfreq is 1.0 by construction and the turn is flagged. The battery is sampled, so the same fixture passes when the model happens to space its JSON and fails when it does not: hw-gate run 33862054891 failed the qwen3.8:27b-mq4-xt battery on a correct (recall 6/6) 17-token JSON answer, and the gate hard-floors on any attractor.
The frequency tests now require a window of at least ATTRACTOR_MIN_WINDOW (8) tokens — repetition needs room to repeat; runaway/empty already cover degenerate short output. gram3 is unchanged (it already needs 6). Self-test gains the short-answer cases and an 8x-repeat that must still trip.
4 tasks
Kaden-Schutt
added a commit
that referenced
this pull request
Sep 4, 2026
Third defect from tonight's ladder, and the one with real teeth: the hardware lanes execute `pr/scripts/serve_harness.py` — the branch's own copy of the instrument that judges it. #682 is a fork PR based on master from before #703, so its harness has no `ATTRACTOR_MIN_WINDOW` guard. Run 33921475093 ran that stale copy and flagged the 3-token answer `Answer: 43` as a token attractor on BOTH lanes, exiting 1 on a fixture whose own rows record finish=stop, gen=9, recall=1/1, attractor false, empty false, runaway false. Two green lanes turned red and the rung hard-floored on evidence that was correct. Two consequences, one worse than the other: - any PR older than a harness fix inherits that harness's bugs, which is a false-failure generator that gets worse as the harness improves - a branch can weaken its own oracle, which is the part that must never be possible in a gate Fixtures, registry, prompts and review.py already resolve against the gate checkout for exactly this reason (see the `_gate_root` comment above the battery-prompt resolution). serve_harness.py and redline_daemon_harness.py now do too, including their existence preconditions. The PR still supplies the binaries under test — only the instrument moved. Test: `test_serve_harness_runs_from_gate_root_not_pr` writes a different harness into each tree and asserts the gate's is the one invoked. 106/106 hw-gate tests pass.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
_token_attractorsplits the visible answer on whitespace and fails the turn whenmaxfreq(first) > 0.50. A one-"word" answer — compact JSON{"name":"Alice","age":34,"city":"Lisbon"},Paris,43— is a single token, somaxfreqis 1.0 by construction and the turn is flagged!ATTRACTOR.The battery is sampled, so the same fixture passes when the model spaces its JSON and fails when it doesn't. hw-gate run 33862054891 (#686 re-run) failed the
qwen3.8:27b-mq4-xtbattery on a correct, recall-6/6, 17-token JSON answer; the same daemon md5 passed the same fixture two hours earlier with{"name": "Alice", ...}. The gate hard-floors on any attractor, so this is a random red on every PR whose battery draws a compact answer.Change
The frequency tests (
uniq,maxfreqon the first/last 128 tokens) now require a window of at leastATTRACTOR_MIN_WINDOW = 8tokens — repetition needs room to repeat in;runaway/emptyalready cover degenerate short output.gram3is unchanged (already needs 6). One function, one constant.Evidence
{"name":"Alice","age":34,"city":"Lisbon"}Parisyes yesthe×8the×40a b c×20 (gram3 loop)The harness self-test (
_self_test_attractor_channels, run at startup) gains the short-answer cases and an 8×-repeat that must still trip; passes.Which surface(s) does this touch?
scripts/serve_harness.pyonly. No Rust, no kernels.Note for the gate:
run.pyexecutes the harness from the PR checkout, so PRs pick this up on rebase.