Skip to content

test(cluster): fix flaky sync hang via lossless watch; adopt store update - #74

Merged
benitogf merged 1 commit into
masterfrom
fix/cluster-test-flake
Jun 18, 2026
Merged

test(cluster): fix flaky sync hang via lossless watch; adopt store update#74
benitogf merged 1 commit into
masterfrom
fix/cluster-test-flake

Conversation

@benitogf

Copy link
Copy Markdown
Owner

Summary

Fixes intermittent cluster-sync misses under heavy load by adopting the store's lossless real-time delivery, and makes the cluster sync test deterministic. Closes #73.

What changed

  • Adopt lossless real-time delivery. The cluster sync test harness opts into the store's lossless watch mode, so every committed write reaches every live subscriber instead of occasionally being dropped when a consumer stalls under load. A guard fails the test immediately if any event is ever dropped. The sync test's own synchronization is unchanged — the dropped delivery, not the test, was the flake (it manifested as a ~1/300 hang under high-iteration -race).
  • Stop a background watch-worker leak. The instance ties its storage watchers to its lifetime, so they stop on shutdown instead of lingering across runs.
  • Adapt to the dependency's removed request gate. The store replaced its built-in request gate with standard router middleware. Permissive test hooks drop to the new default (open), and the one hardened-node probe test moves to middleware.

Verification

  • /flaky-check: 300 cumulative full-suite -race iterations clean against the released dependency (and 540 against the same fix before release), zero dropped events. Full suite green under -race.

Compatibility

  • Picks up a store release that removes the built-in request gate (now middleware-based). Only test code referenced it in this repo.

🤖 Generated with Claude Code

…h; port Audit→Router.Use

Bump ooo to the lossless-watch release. TestClusterSyncLocal/Remote hung
intermittently (~1/300 -race iterations, only under -count) because ooo dropped
a storage→broadcast event when a consumer stalled under load, so a subscriber
missed a delivery and the test's WaitGroup waited forever. The WaitGroup counts
were correct all along — the loss was below them.

FakeServer now enables ooo's LosslessWatch (block instead of drop) with an
OnDroppedEvent guard that fails the test if any event is ever dropped, so every
committed write deterministically reaches every live subscriber. The cluster
sync test's WaitGroup logic is unchanged.

The ooo bump also removes Server.Audit (now gated via Router.Use middleware):
drop the allow-all Audit test hooks (no middleware = allow all by default) and
port the one deny gate (nodehealth ping probe test) to Router.Use. Pass the
instance context to WatchWithCallback, now context-cancellable, so attached
storages' watch workers stop on shutdown instead of leaking.

Verified with /flaky-check: 300 cumulative full-suite -race iterations clean
against the released ooo (540 against the same fix pre-release), zero drops.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

@CBosch101 CBosch101 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Adopts the lossless-watch ooo release to kill the ~1/300 cluster-sync flake and ports the removed Server.Audit gate to router middleware. Mostly mechanical test adaptation; the one production change is sound. Approving.

Verified

  • go build ./... + go vet ./... clean against the bumped ooo; TestClusterSync* and TestPingNodeProbesUnauthedEndpoint pass under -race.
  • instance.go:399 threads i.ctx into WatchWithCallbackctx is set in Setup and cancelled in Shutdown, so attached storages' watch workers now stop on shutdown instead of leaking.
  • Bumped ooo actually provides the API this PR relies on: LosslessWatch/OnDroppedEvent fields and the ctx-first WatchWithCallback signature; Server.Audit is genuinely removed (the dropped allow-all test hooks and the nodehealth_pingnode_test.go Router.Use port match that).
  • Issue #73 acceptance: lossless delivery + OnDroppedEvent guard that fails on any drop, dependency updated, suite builds/passes adapting to the removed gate — all addressed.

The fix is structural (block-instead-of-drop + a guard that fails the test if any event is ever dropped), not statistical, so it targets the documented root cause directly. I ran the cluster/ping tests once under -race, not the full 300-iteration sweep.

@benitogf
benitogf merged commit 07bfbd7 into master Jun 18, 2026
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Cluster sync can miss a real-time update under heavy load

2 participants