Skip to content

fix(evals): publish a referee message when it is delivered, not when it is decided - #1176

Merged
Poytr1 merged 1 commit into
mainfrom
fix/werewolf-eval-referee-publish-race
Aug 17, 2026
Merged

fix(evals): publish a referee message when it is delivered, not when it is decided#1176
Poytr1 merged 1 commit into
mainfrom
fix/werewolf-eval-referee-publish-race

Conversation

@Poytr1

@Poytr1 Poytr1 commented Aug 17, 2026

Copy link
Copy Markdown
Contributor

The flake

evals/test/werewolf.test.tsSCRIPTED BOUNDARY: a seven-player game exhausts the budget inside one 60s window failed roughly 1 run in 7 with expected 20 to be 8, and its expect(latched.length).toBeGreaterThan(0) was flaking too. Not caused by #997assignWerewolfRoles is byte-identical at 7 players before and after it.

Root cause: a message became visible before it was delivered

WerewolfGame.roomBroadcast wrote the referee's post into the provider-visible thread (world.recordThreadMessage) at the moment the message was decided, but its platform ingress only entered the daemon when the next wave was emitted.

The referee decides a phase message from inside applyEffects, which the runner calls mid-cascade — peer turns woken by the effect that ended the phase are still open:

  1. The last day-1 speaker posts; its echo wakes the other six players' turns.
  2. applyEffects sees the order complete → closeDiscussionAndQueueVote() → the VOTE 1. text lands in the thread history now.
  3. Those still-open turns hit the daemon's turn-final context refresh, read VOTE 1. out of the transcript, and vote — on a phase change the daemon never delivered.
  4. Because no VOTE 1. turn was ever admitted, the trusted-human turn that resets the automatic loop-guard counter never happened. The five votes were charged to each player's peer circuit on top of the five discussion wakes → 9 automatic turns → every circuit latched at MAX_AUTOMATIC_TURNS_PER_WINDOW.

Whether step 3 caught an open turn was pure scheduling, so the run forked between two entirely different games: latch-everything / six-round round_limit (~85%) versus play-it-out / two-round completed (~15%). The 20 to be 8 assertion was the healthy branch leaking into a test that pinned the broken one.

The fix

  • roomBroadcast and privateDelivery return thunks; pendingWaves holds (() => GameWave)[]; nextDeliveries() invokes the thunk. The thread row, the world event, and the ingress become one atomic step at emit time, so an open turn can only refresh on messages the daemon actually delivered.
  • Each echo outcome now reports its room, so peer.wake records which loop-guard circuit was charged (a member of two rooms holds two independent budgets).

The test

The SCRIPTED BOUNDARY premise was an artifact of the bug — seven seats never exhaust the budget once every phase arrives as a delivered referee turn. Rewritten as the complement of the twelve-seat test, asserting invariants rather than a magic number:

  • nothing latches and no wake is gated;
  • per room, no player takes MAX_AUTOMATIC_TURNS_PER_WINDOW automatic wakes between two referee posts (imported constant; max observed is 6);
  • only wolves are charged in the den;
  • every speech follows a DAY n. post and every vote follows a VOTE n. post — this fails directly if a referee message ever reaches an open turn early again.

The sequential-day test's "later days are allowed to stall" allowance was explicitly conditioned on the same artifact, so it is tightened to every day completing its order.

Verification

  • 20 consecutive quiet runs of the file: 20/20 pass.
  • 20 more runs alongside six concurrent pnpm eval:collab:contracts passes: 20/20 pass, with all 9 contract-suite passes green (17 files, 122 tests).
  • Before the fix, 20 runs of the seven-seat game produced byte-identical wake tables — the game is now deterministic run to run.
  • eslint and prettier --check clean.

Follow-up (not in this PR)

evals/games/counting.ts's referee-announced variant calls roomBroadcast from applyEffects with the same early-publication shape. Its tests aren't currently sensitive to it, so it is left out of scope and tracked separately.

🤖 Generated with Claude Code

…it is decided

`WerewolfGame.roomBroadcast` wrote the referee's post into the
provider-visible thread the moment the message was DECIDED, while its
platform ingress only entered the daemon with the NEXT wave. The referee
decides a phase message from inside `applyEffects`, which the runner calls
mid-cascade — peer turns woken by the effect that ended the phase are still
open. Those turns could refresh their context onto a message the daemon had
never delivered, act on the phase change, and have the reply charged to their
PEER circuit: the trusted-human turn that resets the automatic loop-guard
budget never happened, so a day's discussion and its votes piled onto one
window and every circuit latched at MAX_AUTOMATIC_TURNS_PER_WINDOW.

Whether an open turn caught the early write was pure scheduling, so a
seven-seat scripted run forked between two whole games — latch-everything /
six-round `round_limit`, or play-it-out / two-round `completed` — and the
SCRIPTED BOUNDARY test flaked roughly one run in seven.

`roomBroadcast` and `privateDelivery` now return thunks and `pendingWaves`
holds them, so `nextDeliveries()` publishes the thread row, the world event,
and the ingress as one atomic step at emit time. Each echo outcome also
reports its room, so `peer.wake` records which circuit was charged.

The SCRIPTED BOUNDARY test pinned the broken branch, and its premise was an
artifact of the bug: seven seats never exhaust the budget once every phase
arrives as a delivered referee turn. Rewritten as the complement of the
twelve-seat test, asserting invariants instead of a magic number — nothing
latches, no wake is gated, no player takes MAX_AUTOMATIC_TURNS_PER_WINDOW
automatic wakes on one room's budget between two referee posts, only wolves
are charged in the den, and every speech follows a DAY post while every vote
follows a VOTE post. That last one fails directly if a referee message ever
reaches an open turn early again. The sequential-day test's "later days are
allowed to stall" allowance was conditioned on the same artifact and is now
tightened to every day completing its order.

Verified with 40 consecutive runs of the file (20 quiet, 20 alongside six
concurrent `eval:collab:contracts` passes) — all green, and the seven-seat
game is now byte-identical run to run.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@Poytr1
Poytr1 merged commit 5076f52 into main Aug 17, 2026
8 checks passed
@Poytr1
Poytr1 deleted the fix/werewolf-eval-referee-publish-race branch August 17, 2026 13:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant