fix(evals): publish a referee message when it is delivered, not when it is decided - #1176
Merged
Merged
Conversation
…it is decided `WerewolfGame.roomBroadcast` wrote the referee's post into the provider-visible thread the moment the message was DECIDED, while its platform ingress only entered the daemon with the NEXT wave. The referee decides a phase message from inside `applyEffects`, which the runner calls mid-cascade — peer turns woken by the effect that ended the phase are still open. Those turns could refresh their context onto a message the daemon had never delivered, act on the phase change, and have the reply charged to their PEER circuit: the trusted-human turn that resets the automatic loop-guard budget never happened, so a day's discussion and its votes piled onto one window and every circuit latched at MAX_AUTOMATIC_TURNS_PER_WINDOW. Whether an open turn caught the early write was pure scheduling, so a seven-seat scripted run forked between two whole games — latch-everything / six-round `round_limit`, or play-it-out / two-round `completed` — and the SCRIPTED BOUNDARY test flaked roughly one run in seven. `roomBroadcast` and `privateDelivery` now return thunks and `pendingWaves` holds them, so `nextDeliveries()` publishes the thread row, the world event, and the ingress as one atomic step at emit time. Each echo outcome also reports its room, so `peer.wake` records which circuit was charged. The SCRIPTED BOUNDARY test pinned the broken branch, and its premise was an artifact of the bug: seven seats never exhaust the budget once every phase arrives as a delivered referee turn. Rewritten as the complement of the twelve-seat test, asserting invariants instead of a magic number — nothing latches, no wake is gated, no player takes MAX_AUTOMATIC_TURNS_PER_WINDOW automatic wakes on one room's budget between two referee posts, only wolves are charged in the den, and every speech follows a DAY post while every vote follows a VOTE post. That last one fails directly if a referee message ever reaches an open turn early again. The sequential-day test's "later days are allowed to stall" allowance was conditioned on the same artifact and is now tightened to every day completing its order. Verified with 40 consecutive runs of the file (20 quiet, 20 alongside six concurrent `eval:collab:contracts` passes) — all green, and the seven-seat game is now byte-identical run to run. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The flake
evals/test/werewolf.test.ts→ SCRIPTED BOUNDARY: a seven-player game exhausts the budget inside one 60s window failed roughly 1 run in 7 withexpected 20 to be 8, and itsexpect(latched.length).toBeGreaterThan(0)was flaking too. Not caused by #997 —assignWerewolfRolesis byte-identical at 7 players before and after it.Root cause: a message became visible before it was delivered
WerewolfGame.roomBroadcastwrote the referee's post into the provider-visible thread (world.recordThreadMessage) at the moment the message was decided, but its platform ingress only entered the daemon when the next wave was emitted.The referee decides a phase message from inside
applyEffects, which the runner calls mid-cascade — peer turns woken by the effect that ended the phase are still open:applyEffectssees the order complete →closeDiscussionAndQueueVote()→ theVOTE 1.text lands in the thread history now.VOTE 1.out of the transcript, and vote — on a phase change the daemon never delivered.VOTE 1.turn was ever admitted, the trusted-human turn that resets the automatic loop-guard counter never happened. The five votes were charged to each player's peer circuit on top of the five discussion wakes → 9 automatic turns → every circuit latched atMAX_AUTOMATIC_TURNS_PER_WINDOW.Whether step 3 caught an open turn was pure scheduling, so the run forked between two entirely different games: latch-everything / six-round
round_limit(~85%) versus play-it-out / two-roundcompleted(~15%). The20 to be 8assertion was the healthy branch leaking into a test that pinned the broken one.The fix
roomBroadcastandprivateDeliveryreturn thunks;pendingWavesholds(() => GameWave)[];nextDeliveries()invokes the thunk. The thread row, the world event, and the ingress become one atomic step at emit time, so an open turn can only refresh on messages the daemon actually delivered.peer.wakerecords which loop-guard circuit was charged (a member of two rooms holds two independent budgets).The test
The SCRIPTED BOUNDARY premise was an artifact of the bug — seven seats never exhaust the budget once every phase arrives as a delivered referee turn. Rewritten as the complement of the twelve-seat test, asserting invariants rather than a magic number:
MAX_AUTOMATIC_TURNS_PER_WINDOWautomatic wakes between two referee posts (imported constant; max observed is 6);DAY n.post and every vote follows aVOTE n.post — this fails directly if a referee message ever reaches an open turn early again.The sequential-day test's "later days are allowed to stall" allowance was explicitly conditioned on the same artifact, so it is tightened to every day completing its order.
Verification
pnpm eval:collab:contractspasses: 20/20 pass, with all 9 contract-suite passes green (17 files, 122 tests).eslintandprettier --checkclean.Follow-up (not in this PR)
evals/games/counting.ts'sreferee-announcedvariant callsroomBroadcastfromapplyEffectswith the same early-publication shape. Its tests aren't currently sensitive to it, so it is left out of scope and tracked separately.🤖 Generated with Claude Code