test(cluster): fix flaky sync hang via lossless watch; adopt store update - #74
Merged
Conversation
…h; port Audit→Router.Use Bump ooo to the lossless-watch release. TestClusterSyncLocal/Remote hung intermittently (~1/300 -race iterations, only under -count) because ooo dropped a storage→broadcast event when a consumer stalled under load, so a subscriber missed a delivery and the test's WaitGroup waited forever. The WaitGroup counts were correct all along — the loss was below them. FakeServer now enables ooo's LosslessWatch (block instead of drop) with an OnDroppedEvent guard that fails the test if any event is ever dropped, so every committed write deterministically reaches every live subscriber. The cluster sync test's WaitGroup logic is unchanged. The ooo bump also removes Server.Audit (now gated via Router.Use middleware): drop the allow-all Audit test hooks (no middleware = allow all by default) and port the one deny gate (nodehealth ping probe test) to Router.Use. Pass the instance context to WatchWithCallback, now context-cancellable, so attached storages' watch workers stop on shutdown instead of leaking. Verified with /flaky-check: 300 cumulative full-suite -race iterations clean against the released ooo (540 against the same fix pre-release), zero drops. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
CBosch101
approved these changes
Jun 17, 2026
CBosch101
left a comment
Contributor
There was a problem hiding this comment.
Adopts the lossless-watch ooo release to kill the ~1/300 cluster-sync flake and ports the removed Server.Audit gate to router middleware. Mostly mechanical test adaptation; the one production change is sound. Approving.
Verified
go build ./...+go vet ./...clean against the bumped ooo;TestClusterSync*andTestPingNodeProbesUnauthedEndpointpass under-race.instance.go:399threadsi.ctxintoWatchWithCallback—ctxis set inSetupand cancelled inShutdown, so attached storages' watch workers now stop on shutdown instead of leaking.- Bumped ooo actually provides the API this PR relies on:
LosslessWatch/OnDroppedEventfields and the ctx-firstWatchWithCallbacksignature;Server.Auditis genuinely removed (the dropped allow-all test hooks and thenodehealth_pingnode_test.goRouter.Useport match that). - Issue #73 acceptance: lossless delivery +
OnDroppedEventguard that fails on any drop, dependency updated, suite builds/passes adapting to the removed gate — all addressed.
The fix is structural (block-instead-of-drop + a guard that fails the test if any event is ever dropped), not statistical, so it targets the documented root cause directly. I ran the cluster/ping tests once under -race, not the full 300-iteration sweep.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes intermittent cluster-sync misses under heavy load by adopting the store's lossless real-time delivery, and makes the cluster sync test deterministic. Closes #73.
What changed
-race).Verification
/flaky-check: 300 cumulative full-suite-raceiterations clean against the released dependency (and 540 against the same fix before release), zero dropped events. Full suite green under-race.Compatibility
🤖 Generated with Claude Code